Research Insight | GenAI in Research: A New Era Demands New Rules
As Generative AI tools like ChatGPT, Claude, and Gemini rapidly enter the research toolkit, a key question emerges: How can researchers use these tools effectively and responsibly without compromising scientific rigor?
Built on large language models (LLMs) and enhanced with retrieval systems and multimodal processors, GenAI tools offer powerful capabilities in text generation, data analysis, coding, and even study design. For marketing researchers, this means faster literature reviews, streamlined survey development, and new opportunities for analyzing data. Surprisingly, however, the people using GenAI with the greatest frequency often understand it the least. GenAI’s performance depends heavily on how it’s used, meaning that effective prompting, careful model selection, and rigorous validation can make the difference between insightful findings and misleading results. Uncritical enthusiasm risks overstating what GenAI can deliver, while premature dismissal overlooks its potential value.
This Journal of Marketing study offers a guide for researchers seeking to integrate GenAI into their experimental and survey research processes. It provides an overview of how GenAI models function, starting with LLMs as the foundation and expanding into the system-level enhancements that improve their capabilities. The researchers then build on these principles to explore the implications for research. Gaining a working understanding of how these systems operate will help scholars use them responsibly and effectively, with the goal of harnessing GenAI’s strengths while mitigating its risks.
The authors created a companion website for this research, questionableresearch.ai, which provides templates, code, and checklists to help researchers get started the right way.
For more Research Insights, click here.
What You Need to Know
- Literature review: Use GenAI to summarize, synthesize, and explore academic work, but always fact-check for hallucinations.
- Research design: Generate and refine survey items, manipulations, and stimuli with AI, using careful validation to ensure construct clarity.
- Study administration: Conduct dynamic, conversational data collection (e.g., interviews, chat-based surveys), with controls for consistency and privacy.
- Data analysis and interpretation: Use GenAI to assist with code generation and exploratory analysis, while verifying final computations using statistical software.
Abstract
Generative Artificial Intelligence (GenAI) tools based on Large Language Models (LLMs) are quickly reshaping how researchers conduct surveys and experiments. From reviewing the literature and designing instruments, to administering studies, coding data, and interpreting results, these tools offer substantial opportunities to improve research productivity and advance methodology. Yet with this potential comes a critical challenge: researchers often use these systems without fully understanding how they work. This article aims to provide a practical guide for effective and responsible GenAI use in primary research. We begin by explaining how GenAI systems operate, highlighting the gap between their intuitive interfaces and the underlying model architectures. We then examine different use cases throughout the research process, both the opportunities and associated risks at each stage. Throughout our review, we provide flexible tips for best practice and rules for effective and responsible GenAI use, particularly in areas pertaining to ensuring the validity of GenAI coded responses. In doing so, we hope to help researchers integrate GenAI into their workflows in a transparent, rigorous, and ethically sound manner. Our accompanying website (questionableresearch.ai) provides supporting materials, including reproducible coding templates in R and SPSS and sample pre-registrations.
Simon J. Blanchard, Nofar Duani, Aaron M. Garvey, Oded Netzer, and Travis Tae Oh, “New Tools, New Rules: A Practical Guide to Effective and Responsible GenAI Use for Surveys and Experiments Research,” Journal of Marketing. doi:10.1177/00222429251349882