Big data enhances sociological research

Posted by Justin Hesser on December 26, 2013

Humans are a social species, and those interactions are being captured by data.

The average person produces just under a terabyte of information per year, and that number is only going up. For perspective's sake, imagine writing out each binary decision—represented by a one or a zero—by hand. If you scribbled out the amount the we all produce yearly, it wouldn't just extend to Saturn, it would extend to Saturn and back 25 times. Represented by sheep, the collective amount of data produced by people annually would fill the universe snugly, without any gaps.

It puts the "big" in Big Data.

All of this communication has an ancillary effect: sociologists, once constrained to wonder what people were saying and extrapolate from limited data, now have access to a greater wealth of first hand knowledge than they could have ever imagined being possible. And that number is growing quickly. It's lead many researchers to strongly consider the role something as fundamentally nonacademic as social media could yield in terms of lasting insight. 

Take, for example, Jon Levin. The Stanford economist performed an investigation into the ways that vendors set prices on popular auction site EBay. By using custom web application development and the access he had to hundreds of thousands of decisions, he was able to parse out several important trends, which confirmed some theories of pricing but exposed some significant errors. By grounding a largely theoretical field in actually human interactions, he was able to improve both substantially. 

For his efforts, Levin was awarded a John Bates Clark Medal, the highest award given to an economist under 40. 

It's not an isolated achievement, either. A research team at Harvard was able to combine IRS data with school district information to map out the long term effects of being matched up with a good teacher during formative years. Not only did they find that it had an effect on college matriculation rates, it also had an impact on income and the neighborhood a student would eventually end up living in. 

This year, Raj Chetty, who led the study, also won a John Bates Clark Medal.

The takeaway here is clear: when focused through a sharp academic lens, what might seem like a sprawl of Big Data can yield some valuable insights.