When it comes to Big Data, the details matter

Posted by Justin Hesser on February 3, 2014

A Connecticut FileMaker developer won't just provide your company with answers to business-related problems. It could also raise some fascinating questions. 

Consider, for example, the outcome of an argument between Stanford biologist Paul Ehrlich and economist Julian Simon. In 1968, Ehrlich, an entomologist by trade who had spent time studying the population density of butterflies, wrote a book called "The Population Bomb." In it, he surmised that, at the rate the Earth was being filled up, humans were headed for a collapse. The environmental movement wholeheartedly embraced his research and used it as a tentpole for their platform. 

Simon, on the other hand, was difficult to convince. He thought that humanity was imminently adaptable, and that we would endure collapse by finding new resources or changing the ways we used the ones we currently have. He offered to back up his rhetoric with a commodities bet: in 1980 they agreed to wager on what the market price for five different metals would be in a decade's time. If the price jumped wildly, it would signal that resources were strained. If not, it would mean that society had successfully navigated population growth. 

Simon won the bet easily. Not only did the price not soar, it actually decreased significantly. 

And yet, Ehrlich remains convinced that his hypothesis was correct, regarding the need for environmental awareness. What was supposed to be an empirically provable fact turned out to lead to further contention, not less. Speaking to Planet Money, Ehrlich doubled down on his original stance, despite the fact that Simon had been proved right in one specific case.

"[Simon] knew absolutely nothing about anything important," he said. "[The bet has] become what's technically known in science as a pain in the [rear]."

There are two valuable lessons in here for managers looking to turn to Big Data. One, that the parameters that you set before you gather data are important.  While both men thought that their commodities bet would be the best way to solve their dispute, that optimism later turned out to be incorrect. 

Secondly, Big Data isn't something you can simply plug in and forget about. Your business should be constantly adapting and altering its strategies, to continue to get the most out of its analytical initiatives. It turns out, a lot can change in a decade.