Registered Replication Reports at AMPPS

Instructions for Authors

Registered Replication Reports (RRRs) combine the results of multiple independently conducted direct replications of an important original study, all of which follow a vetted, accurate protocol. AMPPS seeks proposals for such RRR projects.

The submission and review process for an RRR differs from a standard peer review and involves multiple stages:

1. Authors submit a proposal in which they identify the original study that would be replicated and justify why that study has high replication value. Given the extensive resources for a multi-lab study, only those original studies with the highest replication value can be considered, and they must be feasible for a multi-lab approach. Replication value and suitability of a study for an RRR reflects whether:

(a) The study has been influential in the field

(b) The study is methodologically sound and more recent methods have not superseded those in the original study

(c) The interpretation of the result is unambiguous, and the findings either force a reconsideration of an important theory or establish the foundation for a theoretical position

(d) The study has not yet been the subject of published replications, or published replications have yielded inconsistent estimates of the size of the effect, leading to uncertainty about the size of the effect

(e) Theories of or empirical understanding of the effect would benefit from a more precise estimate of the effect’s size

(f) Other labs likely would be able to join the replication effort (the authors are encouraged to identify other laboratories known to be interested in contributing)

(g) The original study and replications of it would be of interest to the broad readership of AMPPS

Download the Proposal Form

2. This proposal form undergoes an initial review by multiple members of the editorial team. Given our stringent criteria, many proposals will be “desk rejected” at this initial review stage. Those passing this initial screening will be sent out for expert review. This review stage tends to be fairly efficient: reviewers are asked to evaluate the replication value of the original study (using the criteria above) and to gauge the feasibility of the study for our RRR process. In some cases, the review process may lead to a recommendation for an RRR based on a different study of the same topic.

3. If the proposal passes the review process, it will be provisionally accepted. Authors will then develop the protocol, materials, scripts, and code for the study. The editors will contact the original authors to inform them that an RRR project is in development and to ask for their assistance in providing any materials, code, and data to the authors. (In some cases, the authors may already have been in touch with the original authors.) The protocol should be sufficiently complete that a laboratory could follow it from start to finish to conduct the study properly and without assistance in their own laboratory. The authors will also draft a pre-data version of the manuscript with a complete introduction and method section. The paper should have a results section that describes the planned analyses and includes placeholders for the eventual results. Ideally, this draft of the paper will include analysis scripts and figures based on simulated data. The editor will provide the authors with example protocols and pre-data manuscripts that they can use as a guide for this process.

4. The Stage 1 pre-data manuscript should be submitted as a revision to the provisionally accepted proposal. All materials necessary to evaluate the study plan should be provided.

5. The editor will review the materials and may suggest revisions prior to review. Once the materials are ready for review, they will be sent to the original authors and optionally to additional expert reviewers (including statistical consultants if necessary). This review process is intended to be constructive, although papers may be rejected at this stage if the full implementation makes clear that the project is infeasible in ways not obvious during the proposal stage. In most cases, the Stage 1 manuscript will undergo revisions until it can be provisionally accepted.

6. Once the Stage 1 manuscript and materials have been provisionally accepted, AMPPS and APS will put out a call for labs interested in contributing to the project, and the editors will evaluate applications to contribute. The study will then proceed according to the plan laid out in the Stage 1 manuscript. The editors and authors can provide guidance and answer questions from contributing labs during this process.

7. Once data collection is complete, the authors will analyze the data according to the Stage 1 plan, incorporate the data into the Stage 1 manuscript, and add a brief discussion section. This Stage 2 manuscript is a draft of the final RRR paper. The authors submit this Stage 2 manuscript as a revision to the provisionally accepted Stage 1 manuscript. This Stage 2 manuscript is again sent for review to the original authors, previous reviewers, and possibly additional reviewers. The purpose of this stage of review is to evaluate whether the study adhered to the plan of the Stage 1 manuscript and to ensure that the conclusions are supported by the evidence. Reviewers may suggest additional exploratory analyses. The revision process will continue until the paper can be accepted.