Weather is part of our everyday lives. Who doesn’t check the rain radar before heading out, or the weather forecast when planning a weekend away? But where does this data come from, and what is it made of? The answer is a mix of measurements, models and statistics, meaning that the use of weather and climate data can get complex very quickly. This session provides a brief overview of the science behind weather and climate forecasts and provides you with the tools to get started with weather data – even if you aren’t a meteorologist. Learn how to connect weather data to other data sources, how to visualize weather and climate data in an interactive weather dashboard embedded in a Python notebook, and other ways you can use weather data for yourself, from examples using weather APIs, maps, PixieDust and Machine Learning.

1. Weather and Climate Data:
Not Just for Meteorologists
Margriet Groenendijk
PyData @ BI Forum Budapest
15 November 2017

2. Weather and Climate Data:
Not Just for Meteorologists
Margriet Groenendijk
PyData @ BI Forum Budapest
15 November 2017
https://www.slideshare.net/MargrietGroenendijk/presentations

3. Weather
Forecast
@MargrietGr

4. Weather
Forecast
in a
notebook
@MargrietGr

5. Data
@MargrietGr
The Weather
Company API
Get free access here:
https://ibm.com/cloud

6. Jupyter notebooks
@MargrietGr

7. Notebooks in the Cloud
@MargrietGr
https://datascience.ibm.com/
Weather example:
https://github.com/MargrietGroenendijk/weather-demos

8. @MargrietGr
import requests
import json
lat = '51.45’
lon = '-0.17’
line='https://'+username+':'+password+
'@twcservice.mybluemix.net/api/weather/v1/geocode/'+
lat+'/'+lon+'/forecast/intraday/10day.json?&units=m’
r=requests.get(line)
weather = json.loads(r.text)
Get access to the API here: https://console.ng.bluemix.net/

9. @MargrietGr
print(json.dumps(weather, indent=4))
{
"forecasts": [
{
"class": "fod_long_range_intraday",
"clds": 42,
"dow": "Tuesday",
...
"temp": 13,
"wdir": 261,
},
],
"metadata":
"latitude": 51.45,
"longitude": -2.58,
...
}
}

10. @MargrietGr
df['date'] = df['fcst_valid_local'].apply(lambda x:
datetime.strptime(x,'%Y-%m-%dT%H:%M:%S+0200'))
import pandas as pd
from datetime import datetime
df = pd.DataFrame.from_dict(json_data['forecasts'])

11. Some more cleaning up
@MargrietGr
df = df.drop([‘expire_time_gmt’],1)
df = df.rename(index=str, columns={‘pop’: ‘chance of rain’, ‘temp’
: ‘temperature’, ’rh’
: ’humidity’, ‘wspd’: ‘wind speed’})
df.head()

12. https://ibm-watson-data-
lab.github.io/pixiedust/
@MargrietGr

13. Plot with PixieDust
@MargrietGr

14. Plot with
PixieDust
@MargrietGr
display(dfmap)

15. Plot with
PixieDust
@MargrietGr
display(dfmap)

16. Plot with
PixieDust
@MargrietGr
display(dfmap)

17. Weather Forecast PixieApp
@MargrietGr

18. PixieApp published on the web through PixieGateway
@MargrietGr

19. But ...
Where does weather
data come from?
@MargrietGr

20. Observations and Models
@MargrietGr

21. Observations
@MargrietGr
Temperature
Humidity
Windspeed and direction
Air pressure
Rainfall
Radiation
http://www.metoffice.gov.uk/public/weather/climate-
network/#?tab=climateNetwork

22. Historic Weather
@MargrietGr
http://www.metoffice.gov.uk/datapoint/
https://www.wunderground.com/weather/api/
https://business.weather.com/products/the-
weather-company-data-packages
https://climexp.knmi.nl
http://www.ecmwf.int/en/forecasts/datasets

23. But …
I want a map
@MargrietGr

24. netcdf binary files (*.nc)
@MargrietGr

25. netcdf binary files (*.nc)
@MargrietGr

26. @MargrietGr
cfile = 'assets/HadCRUT.4.5.0.0.median.nc'
dataset = Dataset(cfile)
print dataset.data_model
print dataset.variables
Data is here:
https://crudata.uea.ac.uk/cru/data/temperature/
from netCDF4 import Dataset, num2date
import numpy as np
NETCDF4
OrderedDict([(u'latitude', <type 'netCDF4._netCDF4.Variable'>
float32 latitude(latitude)
standard_name: latitude
long_name: latitude
point_spacing: even
units: degrees_north
axis: Y
unlimited dimensions:
ts: days since 1850-1-1 00:00:00
calendar: gregorian
start_year: 1850
...

27. @MargrietGr
matplotlib

28. @MargrietGr
import xarray as xr
import matplotlib.pyplot as plt
import cartopy.crs as ccrs
%matplotlib inline
dataset = xr.open_dataset('data/absolute.nc')
dataset.tem[0].plot()

29. @MargrietGr
ax = plt.axes(projection=ccrs.Mollweide(0, 0))
dataset.tem[0].plot(ax=ax, transform=ccrs.PlateCarree())

30. This was all about measurements
What about forecasts
and predictions?
@MargrietGr

31. Climate Models
@MargrietGr

32. @MargrietGr
All parameters Atmospheric
parameters
Carbon cycle
parameters
Sulphur cycle
and oceanic parameters
Climate Models Experiments
Global
Temperature
anomalies
1850-2100

33. What can I use
weather data for?
@MargrietGr

34. Example
@MargrietGr
Traffic
Collisions
and Weather
www.pexels.com/photo/blur-cars-dew-drops-125510/

35. NYPD Traffic Collisions
@MargrietGr
1,119,577
Collisions since July
2012
https://data.cityofnewyork.us/Public-
Safety/NYPD-Motor-Vehicle-Collisions/h9gi-
nx95

36. Weather
@MargrietGr
Historic
Weather
https://business.weather.com/products/
weather-data-packages

37. Collisions Data
@MargrietGr
Load into Pandas DataFrame

38. Collisions Data
@MargrietGr
Load into Pandas DataFrame

39. Explore Data with PixieDust
@MargrietGr

40. Explore Data with PixieDust
@MargrietGr
Load into Pandas DataFrame

41. Explore Data with PixieDust
@MargrietGr

42. Hypothesis
The number of traffic
collisions is influenced by
the weather
@MargrietGr

43. Combine collisions and weather data
@MargrietGr
times = pd.DatetimeIndex(collisions.Date)
collisions1 =
collisions.groupby([times.year,times.month,times.day,times
.hour]).sum()
collisions1['Hour']=collisions1.index
collisions_hour = collisions1.reset_index(drop=True)
collisions_hour['Date']=collisions_hour['Hour'].apply(lambda x:
datetime(int(x[0]),int(x[1]),int(x[2]),int(x[3])))
merged_df =
pd.merge_asof(collisions_hour.sort_values(by='Date'),
weather.sort_values(by='date'),
left_on='Date',right_on='date',tolerance=pd.Timedelta(’1h'
))

44. Combine collisions and weather data
@MargrietGr

45. Analysis with scikit-learn
@MargrietGr

46. Get the data ready for analysis
@MargrietGr

47. @MargrietGr

48. Additional Data Needed
@MargrietGr
Speed limit
Number of lanes
Traffic volume
Population density
Road quality
Usage by trucks and buses
Pavement rating
Width
https://data.cityofnewyork.us/Transportation/Street-
Pavement-Rating/2cav-chmn
For each collision find
nearest road based on
Latitude and Longitude

49. Thank you!
Margriet Groenendijk
Data Scientist
Developer Advocate
mgroenen@uk.ibm.com
@MargrietGr
Slides
https://www.slideshare.net/MargrietGroenen
dijk/presentations
Blog
https://medium.com/ibm-watson-data-lab
@MargrietGr
IBM Data Science Experience
https://datascience.ibm.com
PixieDust
https://ibm-watson-data-lab.github.io/pixiedust/
Notebooks
https://github.com/ibm-watson-data-lab/python-
notebooks
Weather Data
https://business.weather.com/products/weather-
data-packages
IBM Bluemix
https://console.ng.bluemix.net/

50. @MargrietGr