ARC: Community: ELMAR: Posting

areas: methods: dialog

---

About two years ago, I was a reasonable person who argued that tests of statistical significance were useful in some limited situations. After completing research for "Significance tests harm progress in forecasting" in the International Journal of Forecasting, 23 (2007), 321-327, I have concluded that tests of statistical significance should never be used. Here is the abstract:

I briefly summarize prior research showing that tests of statistical significance are improperly used even in leading scholarly journals. Attempts to educate researchers to avoid pitfalls have had little success. Even when done properly, however, statistical significance tests are of no value. Other researchers have discussed reasons for these failures. I was unable to find empirical evidence to support the use of significance tests under any conditions. I then show that tests of statistical significance are harmful to the development of scientific knowledge because they distract the researcher from the use of proper methods. I illustrate the dangers of significance tests by examining a re-analysis of the M3-Competition. Although the authors of the re-analysis conducted a proper series of statistical tests, they suggested that the original M3-Competition was not justified in concluding that combined forecasts reduce errors, and that the selection of the best method is dependent on the selection of a proper error measure; however, I show that the original conclusions were correct. Authors should avoid tests of statistical significance; instead, they should report on effect sizes, confidence intervals, replications/extensions, and meta-analyses. Practitioners should ignore significance tests and journals should discourage them. http://dx.doi.org/10.1016/j.ijforecast.2007.03.004

The paper is followed by commentaries by Keith Ord, Herman Stekler, and Paul Goodwin, and by my reply: "Statistical significance tests are unnecessary even when properly done and properly interpreted: Reply to commentaries" at http://dx.doi.org/10.1016/j.ijforecast.2007.01.010 This is happy news for practitioners, researchers, and students. On the other hand, it might create anguish among faculty who teach people about statistical significance. The papers are also available at http://jscottarmstrong.com

Scott

 —
J. Scott Armstrong
Professor of Marketing, 747 Huntsman, The Wharton School, U. of PA, Phila, PA 19104
http://www.jscottarmstrong.com
home phone 610 622 6480
Home address: 645 Harper Ave., Drexel Hill, PA 19026
Fax at school: 215 898 2534

---