Seven Selfish Reasons for Preregistration

“Personally, I aim never again to submit for publication a report of a study that was not preregistered.” –D. Stephen Lindsay (2015, p. 1827) in an editorial for Psychological Science

With preregistration, researchers stipulate their hypothesis and analysis plan in advance of data collection, essentially tying their own hands and letting the empirical chips fall where they may (Peirce, 1883). The theoretical advantage of preregistration is that it sharpens the distinction between two complementary but separate stages of scientific inquiry: the stage of hypothesis generation (i.e., exploratory research) and the stage of hypothesis testing (i.e., confirmatory research). By respecting this distinction, researchers inoculate themselves against the pervasive effects of hindsight bias and confirmation bias (e.g., Nuzzo, 2015). Preregistration does not prevent researchers from conducting and presenting exploratory analyses, but it does prevent them from unwittingly presenting an exploratory finding as if it had been confirmatory.

In discussions about the pros and cons of preregistration, it is rarely mentioned that the procedure includes substantial personal benefits and that it generally enhances one’s potential for achieving academic success. Below, we list seven selfish reasons to engage in preregistration. An annotated list of online materials and articles that highlight the benefits and practical feasibility of preregistration can be found here.

1. Preregistration allows you to take credit for your predictions.

Moore (2016) suggests: “Preregister anything you want to get credit for having been able to anticipate.” On datacolada.org, Leif Nelson writes: “In a world of transparent reporting, I choose preregistration as a way to selfishly show off that I predicted the outcome of my study.”

2. Preregistration is exciting.

You have a theory you wish to test. You design an elegant experiment and collect many observations. With preregistration, you cannot tinker with the analysis in a posthoc fashion — at least not in secret. You have asked a clear question, and that means you are more likely to receive a clear answer: Will your theory be confirmed or disconfirmed?

3. Preregistration prevents you from being taken hostage by your own data.

Without preregistration, it can be unclear what the data mean, especially for low-power designs and complex measurement methods. This ambiguity encourages fishing expeditions, and such expeditions can take months to complete. Who does not know the hapless grad student who has turned the data inside out, applying a host of alternative analysis options (kindly suggested by various faculty members at the weekly lab meeting) all with the purpose of producing “the goods” (a publishable result)? Because such results are obtained by cherry-picking, the entire enterprise is as useful as digging a hole and filling it back up.

4. Preregistration is easy.

You can easily preregister your experiments online, for instance on the Open Science Framework or on AsPredicted.

5. Preregistration builds your reputation.

When you preregister your experiments, you signal confidence. A strong anvil need not fear the hammer; similarly, you are not afraid to submit your theories to an unambiguous test. Preregistration also signals that you wish to conduct science in a transparent way and that you have done everything in your power to shield yourself from hindsight bias and confirmation bias.

6. Preregistration allows you to have manuscripts accepted “in principle“ regardless of how the results pan out.

This benefit of preregistration can be obtained when you submit your preregistration proposal to a journal that offers Chris Chambers’s “Registered Report,” or RR, format. After a preregistration proposal has been approved, you obtain “in-principle acceptance,” and eventual publication does not depend on the results. Another advantage of the RR format is that it allows you to improve the preregistration proposal based on the reviewers’ comments — that is, prior to data collection and before it’s “too late.”

7. Preregistration can shield you from posthoc critique.

Without preregistration, when your empirical result is not to the reviewers’ liking, they will always be able to find a “crucial” flaw in your experiment — a hidden moderator, an inadequate manipulation check, an unrepresentative population, a subtly different instruction, a lack of power, and so on and so forth. Nosek and Lakens (2014, p. 138) termed this “critiquing after the results are known” (CARKing), and they point out that “[m]otivated reasoning makes it easy to generate stories for why results differed from expectations.” Preregistration can protect you from CARKing, but only if the reviewers have signed off on the preregistration proposal — a procedure that is an integral component of the RR format explained above.

Additional arguments in favor of preregistration can be brought to bear. Spirited discussions with senior people in the field, however, suggest to us that experience trumps logic.¹ Therefore, we end by calling on psychological scientists to give preregistration a try — you may discover that there is little to lose and much to gain.² œ

To learn about how Psychological Science is supporting preregistration, click here.

Eric-Jan Wagenmakers will speak at the 2017 APS Annual Convention, May 25–28, 2017, in Boston, Massachusetts.

¹    This is reminiscent of the seaweed salad that we once enjoyed at a Japanese restaurant in the outskirts of Detroit. Yes, that salad may have looked like green worms drowned in snot, but the taste was marvellous.

²    Supplemental materials are available on the Open Science Framework.

References

Lindsay, D. S. (2015). Replication in psychological science. Psychological Science, 26, 1827–1832.

Moore, D. A. (2016). Preregister if you want to. American Psychologist, 71, 238–239.

Nosek, B. A., & Lakens, D. (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45, 137–141.

Nuzzo, R. (2015). How scientists fool themselves — and how they can stop. Nature, 526, 182–185.

Peirce, C. S. (1883). A theory of probable inference. In C. S. Peirce (Ed.), Studies in Logic, pp. 126–181. Boston: Little & Brown