Reproducible Research at AMA ART Forum

For ART Forum presentations, the committee prefers talks to be fully reproducible, i.e., built via markdown, with shared data and code. At a minimum, a talk should be partially reproducible, with shared code and data (simulated parallel data is permitted) that demonstrate the exact method. Specifically, a practitioner should be able to run an exact script from start to finish without errors, recreating the principal results in the talk, with no need to edit anything other than the location of the data file.

What do we mean by reproducible research?
In the marketing world, research can be presented in a variety of forms. One way to classify research is to identify the extent to which other researchers can independently reproduce the results of the study. The ability to reproduce the results requires that the original research team provide the appropriate code and data used to determine the original results. The following describes types of research that vary depending upon their reproducibility:

Not Reproducible:
Typically, marketing papers and conference talks/panels do not provide reproducible research. Even upon publication, these forms of research are not required to provide either the original data or the code to reproduce the results. Instead, results are copied and pasted manually into the published result. The code and/or data is sometimes available upon request to the authors or, in some instances of peer-review, shared only with reviewers. Therefore, without the code and data it is difficult or impossible for practitioners to apply the methods to their work.

Partially Reproducible:

Published papers, particularly those in peer-reviewed journals, include some relevant data and code to demonstrate the results. However, the paper or talk is written separately and results are included by manual means which can introduce human error. While practitioners may be able to reproduce the results, it still may remain unclear how the results from code are used in the paper. Example: Indranil Goswami and Oleg Urminsky (2016) When Should the Ask Be a Nudge? The Effect of Default Amounts on Charitable Donations. Journal of Marketing Research: October 2016, Vol. 53, No. 5, pp. 829-846 (see supplemental online files).

Fully Reproducible:
Although relatively common in statistics and computer science, fully reproducible research is a rarity in marketing. Published papers and public talks includes the exact code and data such that the results in the paper are automatically generated; charts, tables, and statistical estimates are generated through code without manual copying or pasting of results. Practitioners can recreate the exact published paper by accessing a "markdown" version with included code. Example: EMD Feit and C Chapman (2015), Choice Modeling, Instructor Content for R for Marketing Research and Analytics (markdown file, which automatically builds this resulting presentation).​

Become a Member
Access our innovative members-only resources and tools to further your marketing practice.