Rethinking Type S and Type M Errors

Abstract

Gelman and Carlin introduced Type S (sign) and Type M (magnitude) errors to highlight the possibility that statistically significant results in published articles are misleading. Although these concepts have been proposed to be useful both when designing a study (prospective) and when evaluating results (retroactive), we argue that these statistics do not facilitate the proper design of studies or the meaningful interpretation of results. Type S errors are a response to the criticism of testing against a point null of exactly zero in contexts in which true zero effects are implausible. Testing against a minimum effect while controlling the Type 1 error rate provides a more coherent and practically useful alternative. Type M errors warn against effect-size inflation after selectively reporting significant results, but we argue that statistical indices such as the critical effect size or bias-adjusted effect size are preferable approaches. We do believe that Type S and Type M errors can be valuable in statistics education, in which the principles of error control are explained, and in the discussion section of studies that fail to follow good research practices. Overall, we argue that their use cases are more limited than is currently recognized and that alternative solutions deserve greater attention.