How to collect and analyze data from 100,000 weather stations
Want to know what it's like to analyze massive amounts of data under pressure? Talk to Bryson Koehler, CIO of The Weather Company (which owns the Weather Channel), who must interpret data sets from around the world to predict something as volatile as the weather.
If you want to understand what it takes to collect, track and analyze reams of data, just check the weather. There are constant fluctuations, scores of data points and intense interest from all over the planet. Analyze the data correctly and someone in the state of Washington knows whether or not to wear a raincoat. Do it poorly and there might be a massive traffic pileup from people driving too fast on slick roads.
Bryson Koehler understands this dynamic. As CIO of The Weather Company, he's charged with increasing the accuracy of weather forecasting for the various entities the company owns, which include the Weather Channel and the Weather Underground mobile app.
The app in particular uses a massive personal sensor network to increase accuracy. Even a smartphone can be a basic weather station: The Weather Company uses algorithms that can determine the outside temperature for that user based on what the phone is reporting.
There are 100,000 sensors sending data worldwide (and 40,000 in the United States alone). Understandably, processing the data is no easy task.
"Some of the data is interesting – such as lightning data or pollen data – and it doesn't always help us create a forecast, but we can tell people who have allergies what to expect,” Koehler says. “Other types of data we get in real time, such as aircraft telemetry data – installations on commercial aircrafts that we bring down in real time to see atmospheric conditions."
Koehler says the flight data is incredibly helpful. It can be used to alert airlines about possible changes in flight plans, or let them know the wear and tear on a plane is not as significant as it might have seemed during a flight. This data can help minimize delays, since the airlines are required to do extra safety checks related to severe weather. The Weather Company can tell if the real-time weather data did not reach as high a threshold as the pilot might have reported.
The analysis is intense. Stations provide data for humidity, barometric pressure, dew point, UV load, rainfall, wind and many other factors. There are billions of reports sent in each month, according to Koehler. The station data is repurposed into a format people can use and understand.
More instruments mean more data
"People can pull up different layers of maps, and they can pull up forecasts from all over the globe,” he says. “In contrast, the National Weather Service in the U.S. has about 3,500 recording stations that they own and operate on behalf of U.S. taxpayers."
It's an interesting dilemma to have such an abundance of data to process. Koehler says that the NWS is one of the world's most “most instrumented” government agencies. Yet, the Weather Company has to deal with many thousands of personal weather stations worldwide. Some of the stations are not easily accessible – they could be in a remote region of Iceland. Some of the weather sensors are as small as a Coke can and some involve an antenna that is three-feet tall.
The Weather Company acts as a “clearinghouse” for this data collection, says Koehler. The company monitors the stations and knows exactly how each one works – that the station is a RainWise product that collects data every second versus a Netatmo station that might not collect as often, for example.