Presidential Column

Why Preregistration Makes Me Nervous

This is a portrait of Susan Goldin-Meadow.I must admit that when I first heard of the effort to get psychological scientists to preregister their studies (that is, to submit to a journal a study’s hypotheses and a plan for how the data will be analyzed before that study has been run), I had a moment of panic. It seemed, on the surface, entirely too regulated for my tastes. I have since calmed down and now see the usefulness of preregistration — indeed, APS has been at the forefront of encouraging preregistration to make our science more transparent and reliable. Manuscripts accepted for publication in Psychological Science are eligible to earn three separate badges designed to promote open science (Eich, 2014). (Editor’s Note: Clinical Psychological Science now offers badges as well. See story on p. 13.). These are

  • the Preregistered badge (for preregistering the design and analysis plan of the reported research and reporting the results as planned);
  • the Open Materials badge (for making the components of the methods needed to reproduce the study publicly available); and
  • the Open Data badge (for making the data needed to reproduce the reported results publicly available).

The badge system has recently been shown to improve open scientific practices.

But my initial reaction to preregistration still needs to be carefully examined and taken seriously, as others may have reacted (and continue to react) in the same way. Casual observation suggests that the Preregistration badge is the least used. For me at least, two concerns continue to keep me up at night.

The first is the fear that preregistration will stifle discovery. Science isn’t just about testing hypotheses — it’s also about discovering hypotheses grounded in phenomena that are worthy of study. Aren’t we supposed to let the data guide us in our exploration? How can we make new discoveries if our studies need to be catalogued before they are run?

The second concern is that preregistration seems like it applies only to certain types of studies — experimental studies done in the lab under controlled conditions. What about observational research, field research, and research with uncommon participants, to name just a few that might not fit neatly into the preregistration script?

This month’s column will be devoted to ruminating about the worry that preregistration will stifle discovery. Next month’s column will focus on the worry that preregistration applies only to a certain type of study.

APS Fellow Paul Rozin, following the late social psychologist Solomon Asch, reminds us that there are stages to conducting science. The first stage is devoted to discovering phenomena, describing them appropriately (i.e., figuring out which aspects of the phenomenon define it and are essential to it), and exploring the robustness and generality of the phenomenon. Only after this step has been taken (and it is not a trivial one) should we move on to exploring causal factors — mechanisms that precede the phenomenon and are involved in bringing it about, and functions that follow the phenomenon and lead to its recurrence.

Preregistration is appropriate for Stage 2 hypothesis-testing studies, but it is hard to reconcile with Stage 1 discovery studies. And according to both Rozin and Asch, psychological science has been too quick to move on to Stage 2 studies. We need to know that a phenomenon is robust and generalizable across conditions and participants (or restricted in interesting ways to a set of conditions and participants) before rushing to try to explain the mechanisms responsible for that phenomenon. Understanding a mechanism is worthwhile only if it is a mechanism that underlies a central aspect of our cognitive, social, and biological selves.

Jean Piaget’s work is a lovely illustration of the Rozin/Asch point that science proceeds first by discovering the phenomena that define us. Piaget’s observations about the steps children go through in learning how to conceptualize the world have defined the field of developmental psychology. The phenomena he identified are robust and generalizable across cultures and cohorts, and they need to be taken seriously by any theory purporting to account for development. Piaget’s own attempts at explaining these phenomena, at uncovering their mechanisms, have been seriously challenged. But the phenomena he described stand as behaviors whose development needs to be explained, and the field is still working on this task. Piaget’s initial studies — which could not have been preregistered and were, moreover, experimentally quite messy (more on this point in next month’s column) — have insured that carrying out subsequent experiments to probe causal mechanisms is worth doing.

Only after a phenomenon has been found to be robust is it worth exploring its causal mechanisms — and for that step, carefully controlled lab studies and statistical methods are essential. A question that thinking about the stages of science brings to mind is whether the statistical methods used in hypothesis-testing studies are appropriate for hypothesis-generating studies. If not, our field may need to invent new statistics to assess the reliability of phenomena discovered during Stage 1.

Preregistration is one way that we can make it clear to ourselves, to our reviewers, and to our readers what our hypotheses are. But we should not take preregistration as a goal unto itself. Before we can register our hypotheses, we need to discover them and to make sure that the mechanisms we seek are for phenomena that are worth explaining.

References

Asch, S. E. (1987). Social psychology. New York, NY: Oxford University Press. (Original work published 1952)

Eich, E. (2014). Business not as usual. Psychological Science, 25, 3–6.

Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., … Nosek, B. A. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS Biology, 14. doi:10.1371/journal.pbio.1002456

Rozin, P. (2001). Social psychology and science: Some lessons from Solomon Asch. Personality and Social Psychology Review, 5, 2–14.

Rozin, P. (2006). Domain denigration and process preference in academic psychology. Perspectives on Psychological Science, 1, 365–376.

Rozin, P. (2009). What kind of empirical research should we publish, fund, and reward? A different perspective. Perspectives on Psychological Science, 4, 435–439.