“Reflections on the Review Process”

Roland T. Rust, University of Maryland

The following is excerpted from my farewell editorial as Editor of IJRM. The complete editorial will be published in Issue#4 2018 in IJRM, and is available open access at

https://www.sciencedirect.com/science/article/pii/S0167811618300491.

After three years as Editor of IJRM (not to mention three years as Editor of the Journal of Marketing and seven years as Editor of the Journal of Service Research) I think it is the appropriate time for me to reflect on the nature of the review process in marketing journals, and how I believe it should change. I would like to highlight several key observations, and also give some recommendations to Editors and future Editors of the leading marketing journals. I have tried to implement these recommendations myself, at all of the journals I have edited. Others can determine the extent to which I have been successful.

Observation #1: There are too many review rounds.

I was having dinner last week with a professor from one of the world’s leading universities who was discussing a paper he has had under review for five years at one of the field’s leading journals. The paper recently received yet another risky revision decision in the fourth round. Such delay, although no doubt well-intentioned on the part of the editor, harms the field, because it slows down the diffusion of knowledge. I would be willing to bet that 90+% of the paper’s current (and eventual) value was present in the initial submission.

To combat this problem, some journals have attempted to institute a 2-round policy. The idea is that the paper should achieve at least conditional acceptance in the second round. Such a policy may have unintended consequences. Given that the top journals all have very high standards for rigor, the only papers that will make it through in two rounds are papers that are already highly-polished in the initial submission, and only “safe” papers that are exploring standard topics in standard ways will have a chance.

Observation #2: Perfection is valued more than timeliness.

The example I gave previously shows the downside of this value system. If it takes 4-5 years to get a paper through the review process, there is no way that the marketing literature can respond in a timely way to fast-moving topics. The Computer Science field combats this by counting proceedings papers more than journal articles, and making fast decisions on those proceedings papers. By marketing’s standards, the CS review process seems “fast and loose.” But at least it is fast, and timely work can surface quickly. By contrast, the marketing literature always seems several years behind.

Observation #3: Rigor is valued more than importance.

Every few years there is an editorial written by greybeards in the field that examines the state of the field. That editorial inevitably concludes that rigor is over-emphasized at the expense of importance. What’s more, this problem is currently getting worse. As most major journals go to co-editor systems, more and more of the field’s most important editorial roles are going to younger and less experienced people. Such people, insecure with their own position and status, want to avoid at all costs any suggestion that they are anything but as rigorous as possible. Unfortunately, younger and less experienced people also tend to have less perspective, and less ability to discern historical trends and the importance of topics. The result is an inexorable shift toward rigor, and a devaluing of importance. We can see this from the table of contents of most major marketing journals. How many of the articles would pass the “grandmother test” of being something that an intelligent non-specialist would find important? In my view, very few articles currently do.

Observation #4: There actually is a rigor vs. relevance tradeoff.

It sounds nice to say that a leading journal should have both, but a simple economic argument demonstrates that the tradeoff is inevitable. Consider a researcher’s time, which can be spent using hours developing rigor, such as studying statistical and methodological skills, or developing substantive knowledge, such as meeting with managers, consulting, and keeping up with the latest trends in the business world. Let R be the hours employed developing rigor, S be the hours employed building substantive knowledge, and T be the total hours available for developing one or the other. Then R+S=T. Assuming that more hours R lead to more rigor and more hours S lead to more relevance, developing more rigor always sacrifices some degree of relevance, and vice versa. The only way to overcome this tradeoff is to have author teams that mix subject experts with methodological experts. That is one reason we are seeing more authors on papers.

Observation #5: True innovation is discouraged.

Truly innovative papers have many impediments. The topic is not yet worked out, so it is not clear yet what research approaches may be appropriate. There are not yet the “mutually forgiven sins” that evolve in a mature research area. For example, an economic model can assume a standard demand model and everything proceeds from there. Not everything is open to attack. In innovative papers, however, everything is open to attack. Typically, the editor will not know who to assign to review the paper, so the attacks are likely to come from multiple angles, some of which may be mutually inconsistent. From Observations 1 and 2, we know that perfection will be harder to achieve for such papers, resulting in either a very long review process, or, more likely, rejection in spite of merit. Truly innovative papers usually attempt to address a problem of importance, which means (consistent with Obervation #3) that the inevitable imperfection (i.e., lack of rigor) will likely be weighted more than the importance of the problem.

Observation #6: Ambitious papers are stifled.

Ambitious papers often try to 1) do more than one thing in the paper, and/or 2) use multiple research methodologies. In the first case, authors are often trying to make a big contribution by accomplishing multiple related goals. Such papers are often stifled by reviewers who expect each one of those goals to be achieved at the level of rigor that would be appropriate if the entire paper focused on one goal, which is impossible for a multiple goal paper to achieve. The reviewer reaction is typically either to reject the paper (“it does not accomplish either goal”) or to suggest that the paper focus on only one goal. Either suggestion risks losing the big picture.

In the second case, authors are trying to get a clearer picture by using different methodologies to investigate different aspects. Often, such papers are reviewed by experts in the different methodologies, who either criticize the parts of the paper where they are not expert, or want the whole paper to focus on their preferred methodology. In either case, the expected level of rigor is typically that which would be appropriate if the entire paper were devoted to one methodology. In such a case the author(s) have little chance.

Recommendation #1: Accept papers quicker.

If there is a timeliness value for ideas, then editors need to recognize that getting that last 1% of rigor may result in a net loss of value. This means that it is often best for the editor to take a stand and accept a paper before everybody on the review team signs off. This means that we need to appoint editors who are secure in their standing in the field, and who are strong enough to make decisions that some AE’s or reviewers may disagree with.

Recommendation #2: Editors need to be the importance police.

Given the tendency of reviewers to simply attack papers and produce a list of problems, the editor needs to counteract the reviewers’ almost exclusive focus on rigor by insisting on problem importance. This can also sometimes mean rejecting an unimportant paper for which the reviewers find few problems. It can also mean giving a paper more of a chance if it is on an important topic. I recommend that papers on important and timely topics should be consciously given more slack with respect to expectations of rigor.

Recommendation #3: Editors need to be willing to overrule the review team.

In my view, a good editor respects the review team, but sees the reviews as advisory. The review decision should not be a vote count. In many cases I have given a paper a second round, even with a unanimous negative appraisal by the review team, if the paper was on a very important and timely topic. I have not overridden a unanimous rejection recommendation in the second or later rounds, because it is incumbent on the author(s) to eventually persuade somebody, but otherwise I have not let a negative reviewer stop a paper, if the paper is important enough, and the negative reviewer has not revealed what I believed to be a fatal flaw. Again, the editor needs to be secure enough to make such determinations.

Recommendation #4: Be more forgiving to papers with multiple related goals or multiple methodologies.

In either case, the big picture is key, and reviewers are notorious for focusing on details and missing the big picture. To counteract this, the Editor must focus on the big picture, and determine whether the importance of the big picture counteracts the inevitable shortcoming related to any individual goal or methodology.

Conclusions

I hope that my time as Editor of IJRM (and also my additional 10 years as Editor of the Journal of Marketing and the Journal of Service Research) have yielded some insights that others in an editorial role in the future may find useful. I have done my best to implement the recommendations that I list above, and I hope that IJRM has been better as a result. I hope we will all work together to make marketing publications more important, and the marketing field as valuable to society as it is capable of being.

---