Estimating the Prevalence of Replication Research Is Challenging, Not Least Because Even a Thorough Database Search Can Be Shallow

Abstract

Replication—duplicating methods to reexamine findings—is the bedrock of science. To survey this foundation for psychological science, Makel et al. (2012) searched 100 psychology journals, finding that all-fields database searches for replicat* returned 2% to 3% of articles. Coding the contents of 500 such articles, they estimated that only 1% to 2% of psychology articles report new replication research. Four studies, reported here, reexamined these results—including two preregistered replications with extension that sampled psychology articles (published pre-2012) without first searching databases. Nearly 20% of these 1,224 articles used “replication” or a variant term to identify replication as a study aim, report duplicating methods, or describe reexamining findings from prior studies. Moreover, authors frequently described reproducing procedures or retesting results without referring to “replication.” My investigation identified possible reasons why previous prevalence estimates for author-identified replication in behavioral research ranged from 1% to 76%, including what discipline, subdiscipline, report sections, study features, or terminology the researchers considered when identifying replications. Chief among these reasons is the fact that around 90% of articles that discuss replication and over 80% of articles with author-identified replications are missed by searching databases for replicat* because databases hold few details per article. Equivalent limitations of database searches may present similar challenges for metascience investigations and research syntheses.