Why Big Data is so Difficult

Don E. Schultz
Marketing News
Current average rating    
Key Takeaways
  • "Generally, we’re social scientists, not engineers. Perhaps with the exception of people who have been working in direct and database marketing, many of us were not trained to deal with massive data sets or sophisticated analytical tools."
  • "We’re also challenged, as marcom people, because Big Data never stops flowing in. It’s perplexing to think that you have your arms around a project, only to find more data coming in over the transom—and you can’t close the transom."
  • "Big Data is messy, it’s dirty and it’s sloppy. It doesn’t fit in the frameworks that we social scientists have developed and codified over time, so when some of it spills out, we’re generally at a loss as to what to do."

Having just returned from an academic conference, where much of the discussion revolved around “Big Data” (there were as many definitions of Big Data as there were participants, plus maybe a few extras), I began to realize why academics and practitioners alike have so much difficulty with the topic. It’s not just that it is new or even that it involves lots of numbers. It’s that Big Data simply confounds and confuses many marketing and communication people because it is so different. 

Generally, we’re social scientists, not engineers. Perhaps with the exception of people who have been working in direct and database marketing, many of us were not trained to deal with massive data sets or sophisticated analytical tools. That’s something that the IT people do for us. The primary tools that we’re accustomed to using have been focus groups, surveys, consumer panel data and the like. We get more than a few hundred responses and we think that we’ve unlocked the mysteries of the universe. Many of us have to learn a new set of tools to be able to move beyond Excel spreadsheets and crosstabs.   

We’re also challenged, as marcom people, because Big Data never stops flowing in. It’s perplexing to think that you have your arms around a project, only to find more data coming in over the transom—and you can’t close the transom. In most traditional marcom research projects, there’s a beginning and an end for data collection. Often, that’s driven by a planning cycle that’s broken down into financial quarters because that’s how budgets are allocated. We’ll do a bit of research in the third quarter because we’ve budgeted for it during that time period on the planning calendar. ​

​​​​​
The following article is available exclusively to Marketing News Subscribers and Members.

     

Author Bio:

Schultz headshot
Don E. Schultz
Don E. Schultz is a professor (emeritus-in-service) of integrated marketing communications at Northwestern University in Evanston, Ill. schultz@northwestern.edu