Seven Selfish Reasons for Preregistration

“Personally, I aim never again to submit for publication a report of a study that was not preregistered.” –D. Stephen Lindsay (2015, p. 1827) in an editorial for Psychological Science

With preregistration, researchers stipulate their hypothesis and analysis plan in advance of data collection, essentially tying their own hands and letting the empirical chips fall where they may (Peirce, 1883). The theoretical advantage of preregistration is that it sharpens the distinction between two complementary but separate stages of scientific inquiry: the stage of hypothesis generation (i.e., exploratory research) and the stage of hypothesis testing (i.e., confirmatory research). By respecting this distinction, researchers inoculate themselves against the pervasive effects of hindsight bias and confirmation bias (e.g., Nuzzo, 2015). Preregistration does not prevent researchers from conducting and presenting exploratory analyses, but it does prevent them from unwittingly presenting an exploratory finding as if it had been confirmatory.

In discussions about the pros and cons of preregistration, it is rarely mentioned that the procedure includes substantial personal benefits and that it generally enhances one’s potential for achieving academic success. Below, we list seven selfish reasons to engage in preregistration. An annotated list of online materials and articles that highlight the benefits and practical feasibility of preregistration can be found here.

1. Preregistration allows you to take credit for your predictions.

Moore (2016) suggests: “Preregister anything you want to get credit for having been able to anticipate.” On datacolada.org, Leif Nelson writes: “In a world of transparent reporting, I choose preregistration as a way to selfishly show off that I predicted the outcome of my study.”

2. Preregistration is exciting.

You have a theory you wish to test. You design an elegant experiment and collect many observations. With preregistration, you cannot tinker with the analysis in a posthoc fashion — at least not in secret. You have asked a clear question, and that means you are more likely to receive a clear answer: Will your theory be confirmed or disconfirmed?

3. Preregistration prevents you from being taken hostage by your own data.

Without preregistration, it can be unclear what the data mean, especially for low-power designs and complex measurement methods. This ambiguity encourages fishing expeditions, and such expeditions can take months to complete. Who does not know the hapless grad student who has turned the data inside out, applying a host of alternative analysis options (kindly suggested by various faculty members at the weekly lab meeting) all with the purpose of producing “the goods” (a publishable result)? Because such results are obtained by cherry-picking, the entire enterprise is as useful as digging a hole and filling it back up.

4. Preregistration is easy.

You can easily preregister your experiments online, for instance on the Open Science Framework or on AsPredicted.

5. Preregistration builds your reputation.

When you preregister your experiments, you signal confidence. A strong anvil need not fear the hammer; similarly, you are not afraid to submit your theories to an unambiguous test. Preregistration also signals that you wish to conduct science in a transparent way and that you have done everything in your power to shield yourself from hindsight bias and confirmation bias.

6. Preregistration allows you to have manuscripts accepted “in principle“ regardless of how the results pan out.

This benefit of preregistration can be obtained when you submit your preregistration proposal to a journal that offers Chris Chambers’s “Registered Report,” or RR, format. After a preregistration proposal has been approved, you obtain “in-principle acceptance,” and eventual publication does not depend on the results. Another advantage of the RR format is that it allows you to improve the preregistration proposal based on the reviewers’ comments — that is, prior to data collection and before it’s “too late.”

7. Preregistration can shield you from posthoc critique.

Without preregistration, when your empirical result is not to the reviewers’ liking, they will always be able to find a “crucial” flaw in your experiment — a hidden moderator, an inadequate manipulation check, an unrepresentative population, a subtly different instruction, a lack of power, and so on and so forth. Nosek and Lakens (2014, p. 138) termed this “critiquing after the results are known” (CARKing), and they point out that “[m]otivated reasoning makes it easy to generate stories for why results differed from expectations.” Preregistration can protect you from CARKing, but only if the reviewers have signed off on the preregistration proposal — a procedure that is an integral component of the RR format explained above.

Additional arguments in favor of preregistration can be brought to bear. Spirited discussions with senior people in the field, however, suggest to us that experience trumps logic.1 Therefore, we end by calling on psychological scientists to give preregistration a try — you may discover that there is little to lose and much to gain.2 œ

To learn about how Psychological Science is supporting preregistration, click here.

Eric-Jan Wagenmakers will speak at the 2017 APS Annual Convention, May 25–28, 2017, in Boston, Massachusetts.

1    This is reminiscent of the seaweed salad that we once enjoyed at a Japanese restaurant in the outskirts of Detroit. Yes, that salad may have looked like green worms drowned in snot, but the taste was marvellous.

2    Supplemental materials are available on the Open Science Framework.

postersevenselfishreasons

References

Lindsay, D. S. (2015). Replication in psychological science. Psychological Science, 26, 1827–1832.

Moore, D. A. (2016). Preregister if you want to. American Psychologist, 71, 238–239.

Nosek, B. A., & Lakens, D. (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45, 137–141.

Nuzzo, R. (2015). How scientists fool themselves — and how they can stop. Nature, 526, 182–185.

Peirce, C. S. (1883). A theory of probable inference. In C. S. Peirce (Ed.), Studies in Logic, pp. 126–181. Boston: Little & Brown

Comments

Bravo! Thanks for leading with a quote from my editorial; I’ve not 100% lived up to that pledge but I’m working on it and certainly everything going forward it preregistered.

One small tweak to your excellent piece is to clarify that preregistration does not necessarily involve submitting the prereg plan to a journal. Most preregs are not submitted to journals. E.g., Psych Science gives badges for preregistration but does not handled Registered Reports per se.

BTW, APS is launching a new journal called Advances in Methodologies and Practices in Psychological Science that will among other things publish RRs and RRRs.

Dan Simons, Scott Lilienfeld, and I have a manuscript extolling prereg in press for the December Observer.

Steve

Steve, thanks for your comment. Indeed, you can preregister also if you do not submit a registered report, for example at osf. This self-preregistration is of course recommendable when a registered report is no option. However, it is much less selfish!
(Of course, self-preregistration also has a selfish benefit in that it forces you to think very well about your hypotheses and analysis before you collect your data.)

I like all of this except for point 1. It says that if you pregistered prediction is disconfirmed by your data, your reputation will suffer. That is not how we want things to be. What matters is whether your theory is falsifiable. That’s more important than whether your theory is true.

Might be of interest to some. I recently gave this lecture to graduate students at University of Sussex where I discuss pre-registration – and how I think the strongest argument against pre-registration is one actually in favour of it:
https://www.youtube.com/watch?v=jR1llLksfZg&t=1836s

Leave a Comment

Your email address will not be published.
In the interest of transparency, we do not accept anonymous comments.
Required fields are marked*