Textbook Analysis Uncovers Erroneous Explanations of Statistical Significance

For nearly 90 years, null-hypothesis significance testing (NHST) has been the most widely used statistical approach in psychological science. Methodologists have long warned that researchers regularly misinterpret and misapply the technique, however, and a group of psychological scientists in Canada believe they may have traced these misunderstandings back to the source: introductory psychology curriculum.

In an analysis of 30 introductory psychology textbooks, including some of the best-selling volumes in North America, scientists based at the University of Guelph found that the vast majority defined or explained statistical significance inaccurately. Many future psychological scientists get their first exposure to statistical methods in those introductory volumes, the researchers emphasize in an article for Advances in Methods and Practices in Psychological Science.

“Psychology researchers may be learning to interpret statistical significance incorrectly quite early in their careers and are likely learning incorrect interpretations in their classes, from their instructors,” write researchers Scott A. Cassidy, Ralitza Dimova, Benjamin Giguère, Jeffrey R. Spence, and David J. Stanley.

In their analysis, the scientists coded the definitions and explanations of statistical significance in each of the 30 textbooks, with a particular focus on common fallacies. They found that 89% of the books that covered statistical significance defined or explained it incorrectly. The most common misconception identified in the books was the odds-against-chance fallacy (defining statistical significance as a less than a 5% likelihood of a result being random.)

The research team observed that in many cases the textbooks could be improved by simply indicating that NHST begins by assuming the null hypothesis is true. Additionally, authors could be encouraged to remove discussions of statistical significance from their books, and simply discuss findings in general terms. An introduction to NHST could be delayed until students’ first statistics course, they suggest.

“Overall, these results suggest that students’ misinterpretations of statistical significance may not be the result of their failing to remember the correct interpretation they were taught,” the scientists write. “Instead, students may be accurately recalling incorrect pedagogy.”

Reference

Cassidy, S.A., Dimova, R., Giguère, B., Spence, J.R., Stanley, D.J. (2019). Failing grade: 89% of introduction-to-psychology textbooks that define or explain statistical significance do so incorrectly. Advances in Methods and Practices in Psychological Science, doi.org/10.1177/2515245919858072

Comments

Maybe it’s time for us all to go Bayesian. Students have an easier time grasping the likelihood ratio of Bayes Factors.

It’s the same place they learn that squaring r somehow adds useful information.

I suspect that many undergraduate students simply aren’t intelligent enough to keep the proper definition straight, even when it’s given to them.

Hello.
How interesting. I am a UK A level Psychology and EPQ tutor so teach research methods constantly. I would very much like to read the full article and look at the UK text books in relationship to your findings.
I am currently writing my PhD proposal so this could impact heavily on my own results
Many thanks
Kerry


APS regularly opens certain online articles for discussion on our website. Effective February 2021, you must be a logged-in APS member to post comments. By posting a comment, you agree to our Community Guidelines and the display of your profile information, including your name and affiliation. Any opinions, findings, conclusions, or recommendations present in article comments are those of the writers and do not necessarily reflect the views of APS or the article’s author. For more information, please see our Community Guidelines.

Please login with your APS account to comment.