Golden Rule of Forecasting needs help

In their new paper, Scott Armstrong, Kesten Green, and Andreas Graefe present a unifying theory of forecasting in the form of a Golden Rule of Forecasting. The rule is to be conservative. Being conservative in forecasting means applying all cumulative knowledge about the forecasting problem and using evidence-based methods.

The paper provides simple and easily understood guidelines on when and how to use the Golden Rule in the form of a checklist. Forecasters can use the checklist to design and to audit their forecasting procedures. Non-experts can use it to assess whether the forecasting procedures for a given project are credible.

The authors tested the 28 Golden Rule guidelines by reviewing published experimental evidence. They found 150 relevant experiments, and the findings all supported the Golden Rule. The average forecast error reduction from using a single guideline was 28%.

Violations of the Golden Rule, the authors conclude, are responsible for the lack of improvement in forecast accuracy in practice over the past half-century, despite the enormous improvements in forecasting methods over that time.

Why, then, are the Golden Rule guidelines violated? One obvious reason is the effects of political pressures on forecasters to give the client what she wants. The desire to please can lead forecasters to be selective of prior knowledge and data, to use unvalidated methods, to adjust forecasts from valid methods, or to ignore validated methods because of a claim that “things are different now.” Another reason is the belief that sophisticated statistical methods applied to ”big data” are sufficient, and so the forecaster ignores cumulative knowledge.

The authors are looking for relevant studies that they have overlooked in their working paper. They are particularly interested to learn of experimental findings that might challenge the Golden Rule. They ask that you check the references in the paper to see whether any of your own experimental studies, or others with which you are familiar, were overlooked.

The authors are also seeking more experts to review the paper before it is published in the Journal of Business Research. They will acknowledge reviewers, so this is an opportunity to share in the adulation – or blame –for the paper.

A copy of the paper is available here.

Please contact Scott at armstrong@wharton.upenn.edu, Kesten at kesten.green@unisa.edu.au, or Andreas at a.graefe@lmu.de if you are able to help. -J. Scott Armstrong