Weather channel leans on big data

Posted by Justin Hesser on November 26, 2013

When it comes to information, weather forecasting is inundated. Not only are more data points being added every day, they're coming from around the world, interacting in a highly delicate pattern. A butterfly flapping its wings, the old saying goes, can set off a chain of reactions that ends in a tornado. 

So when it comes to large scale analysis, the field is a perfect fit.

"Weather is the original big data application," says Bryson Koehler, executive VP and CIO at the Weather Company. "When mainframes first came about, one of the first applications was a weather forecasting model."

Today, his company processes some 20 terabytes of data every day, resulting in what Koehler boasts are the world's most prescient forecasts. However, not content with simply being in poll position, they are in the process of implementing an entirely new platform that will bring the company's analytic capabilities to new levels. 

Currently, the Weather Company uses a structure that comprises a collection of applications and 13 data centers. While mainframes aren't currently part of their plan, a wide range of other databases are, including including MySQL, Microsoft SQL Server, Cassandra, MongoDB and PostgreSQL. This melange allows them to capture 2.2 million points of information every 15 minutes, from centers globally.

The new system, however, will be built on Basho's Riak NoSQL database and run on Amazon's cloud, upgrades which will allow for significantly faster and easier processing. Use of Amazon's cloud allows Koehler to move to what he describes as a "infrastructure-as-a-service model." The new platform splits up the world (from land, to sea, to ice caps) into 30,000 chunks of four square miles each, whose weather can be predicted even weeks in advance.

For a brand as expansive as The Weather company, this raw computational power is a big boon. The parent behind brands like Weather Channel, WeatherFX, Weather Underground and Intellicast, hundreds of thousands of customers rely on the information it provides, including 30 airlines. In all, it receives billions of computer-based requests daily, and they all need to be delivered quickly. 

Even if you're not responsible for delivering weather information to large groups of people, you might still want to be able to track patterns, especially if you farm or garden. Filemaker Consultants can help you to create a custom database that will organize all of your weather information, along with any other variables that might pop up.