Does Market Research Need a Makeover?

Gordon Wyner
Marketing News
Current average rating    
Key Takeaways

What? The tools for knowing customers are shifting toward observational data.

So what? While observational data is strong on realism, it can be noisy and hard to interpret.

Now what? Take steps to check data-driven conclusions against other ways of knowing such as experiments and surveys.

​June 1, 2017

As sources of data increase and the window between data gathering and insight generation decreases, market research’s legacy methods must evolve

The future of market research is uncertain due to ongoing changes within marketing and the surrounding business environment. Research ought to continue to play an important role, but it might be quite different in the evolving marketing ecosystem with the increasingly complex mix of marketers, media, service providers, agencies, data, tools and technologies. Rather than being a separate activity, standing outside of the daily operation of the business, research may become more embedded within the business itself.

The design of market research—the blueprint for data collection, measurement and analysis—provides a useful lens to frame the possibilities for the discipline. Three of the most important types of designs are the survey, observation and experiment.

The popular business media might give the impression that Big Data (gathered through observation rather than surveys or experiments) will become bigger and drown out other approaches. There are reasons to question the desirability and inevitability of this outcome. Trends in current practices with each of these designs give some indication of their prospects in the future of market research. 


A key strength of the survey is its accurate representation of a population through random sampling selection. Other tools focus on narrower (e.g., current customers) and even sometimes artificially drawn populations (e.g., laboratory subjects). Surveys also enable measurement of a wide range of perceptions, intentions, attitudes and behavior that are not easily measured in other ways. However, the validity of measurement is always a challenge.

The random probability sample has faded from use and been replaced by data-driven methods of “engineering” sample representativeness to achieve a predetermined balance. Declining completion rates have raised questions about the soundness of the new approaches. Validation tools, such as linking surveys to actual in-market behaviors, provide important support for specific applications such as ad testing.

User-friendly questionnaire-building tools have enabled broader deployment of the technique by non-specialists. The result is not always good, such as when customer satisfaction surveys are conducted by sales and service representatives with an obvious, sometimes explicitly stated, vested interest in the outcome. Significant parts of the survey process can be conducted by robots.

Continuation of these trends would lead to a mix of positive and negative outcomes. Once the mainstay of market research, the survey is no longer the preeminent tool.


Observation once required substantial human effort to see consumers in their “natural habitat.” The big shift is to observation of many behaviors via computer-controlled sensors.

The observation method is valued for its realism, requiring less intrusion on consumers than survey questionnaires and creating less opportunity for imposing biases through question construction. Passively observed behavior is expected to be a more accurate reflection than other methods.

Observation generates a fast-growing source of data that can be used for research purposes. Consumer products increasingly have sensors that capture behavior in non-obtrusive ways—in cars, homes and appliances. Service interactions are routinely captured as data in doctors’ offices, phone calls, bank interactions and just about every place a consumer makes purchases with credit, debit or other electronic payment methods. Add to these examples the spreading array of apps that perform specific functions for the consumer—such as comparison shopping, making recommendations and issuing directions—all while generating time-specific data in the form of shopper journeys, consideration sets, reviews, opinions and locations.

What makes this cornucopia of data useful as marketing information is the technology to capture and organize it and perform advanced, real-time analysis. The depth and granularity of information on consumers goes well beyond what could be imagined in the prime of survey methods. Traditional predictive analytics are being preempted by artificial intelligence capabilities that attempt to anticipate and deliver against consumer requirements, for example refilling recurring purchases of household goods based on inventory, usage rate and available offers; dynamically pricing offers based on real-time demand (e.g., Uber) or managing your health via real-time feedback on physiological metrics (e.g., Fitbit).

The fact that early applications of these technologies don’t always operate smoothly doesn’t diminish their potential to improve over time as more data is gathered and software is upgraded. The 2017 Marketing Science Institute conference co-sponsored by UCLA, “Harnessing Marketing Analytics for Business Impact,” featured multiple instances of this pattern of increasing ability of analytics to help operate a business at scale. The abundance of data and tools put great pressure on marketing to perform. At the same time the obstacles to success are still substantial due to the size and complexity of the data and the inherent unpredictability of some of the phenomena of interest.

Ironically, as pointed out by UCLA Professor Randy Bucklin, continued improvement and success in analytics could lead to a reduced need for human researchers and analysts due to automation. There’s no specific prediction of when this might happen, but the direction seems clear.


Experimental methods have been available for a long time. However they’ve been limited in marketing applications, possibly due to the difficulty and resource requirements to build controlled environments. The spread of digital technology to more aspects of life creates more ways to alter the environment via software for testing purposes. A test market need not be a physical or geographic location but a virtually defined group that is randomly assigned to alternative marketing such as ads, product concepts and prices. The vast volumes of traffic, purchase and consumption that flow through the internet create greater opportunities for systematically varying the relevant elements of the ecosystem.

Experiments have the advantage of isolating cause-and-effect relationships, thanks to randomization of independent variables. Other methods can only approach causality via assumptions and advanced analytics. Experiments have the advantage over other methods for testing phenomena, such as marketing stimuli, which don’t currently exist in the market. There is potential to avoid limitations of just improving past performance within existing business models. Experimental research is limited by the creativity of the researchers to develop new concepts and to create reasonable approximations to test in market.

Experimentation can control the noise that sometimes clouds inferences from analyses of observational data. It has the potential to correct for serious biases that can occur. In the case of advertising testing, researchers have developed creative ways to statistically adjust groups of people who have become exposed via natural (not randomly controlled) processes. They have refined the matching of respondents in exposed versus non-exposed groups to achieve better results. Yet, direct comparisons between methods show an alarming amount of variability in outcomes. Research is needed to explain the reasons and ideally lead to improved designs. Experimentation can serve as a “check and balance” on long-standing but possibly biased practices that estimate impact via analysis of observational data.

There is renewed interest in experimentation in marketing practice, still mostly based on A/B or test/control designs. More sophisticated designs can magnify the impact and ROI of changes that are introduced into the market. For example, multi-factor designs that systematically alter many variables at once and response surface designs that lead to optimizing outcomes (e.g., usage, revenue and profit) can be even more powerful than simple designs.

Technology may dictate the nature and amount of growth in the various methods of market research. Its influence on marketing-related behaviors almost ensures that research will follow its lead. It seems inevitable that observational data will increase its share of marketing research compared to surveys and experiments. The use cases for alternatives to digital observational data will have to prove that other methods produce more accurate results and that the impact translates to better business outcomes.

Recommended For You:
AMA PCM Digital Marketing Exam Are Your Insights Based on Hypothetical Research or Grounded in Actual Consumer Needs? Seven Experts on Marketing Problem Four: Generating and Using Insight to Shape Marketing Practice

AMA PCM Digital Marketing Exam

Are Your Insights Based on Hypothetical Research or Grounded in Actual Consumer Needs?

Seven Experts on Marketing Problem Four: Generating and Using Insight to Shape Marketing Practice


 Sign Up for Marketing News Weekly

Get the best marketing thought leadership delivered directly to your inbox!

Author Bio:

Gordon Wyner
Gordon Wyner is research director at the Marketing Science Institute.
Add A Comment :

Become a Member
Access our innovative members-only resources and tools to further your marketing practice.