Registered Reports at AMPPS

Instructions for Authors

Registered Reports are a new article type in which the primary review stage occurs prior to data collection, with provisional acceptance meaning that the eventual study will be published regardless of the outcome (provided that it meets pre-specified manipulation checks and positive controls). In AMPPS, registered reports can take many forms: Multi-lab original studies, adversarial collaborations, consortium studies, etc. Note that multi-lab direct replications of an original study should be submitted as the specialized Registered Replication Reports article type rather than as a Registered Report. Our guidelines for Registered Reports draw upon the more extensive guidelines provided at Open Science Framework.

AMPPS Registered Reports generally will report empirical work using a new design rather than a replication, although registered reports may involve replications of an original study with an improved or novel design. Multi-laboratory studies are encouraged, and authors can submit registered reports for secondary data analysis provided that they can supply evidence to confirm that they had no prior access to the data (e.g., a letter from an independent gatekeeper).

Registered reports involve a 2-stage submission process.

1. The Stage 1 manuscript (pre-data collection) includes the following sections:

• Introduction: A review of the relevant literature that motivates the research question and a full description of the experimental aims and hypotheses. Once the Stage 1 manuscript has been provisionally accepted, the substance of the introduction may not be changed.
• Preliminary data/studies: Authors may include pilot data or a preliminary study in the manuscript. The manuscript should make clear that those data are not part of the registered plan.
• Methods: The methods section should include a full description of the sample characteristics and inclusion criteria. It should describe the materials and procedures in sufficient detail that another researcher could conduct the study exactly as intended without further guidance. Ideally, authors should provide any code or materials necessary to conduct the study, allowing the editors and reviewers to assess what participants will do in the study. The method section should justify for the planned sample size (e.g., an a priori power analysis if using null hypothesis significance testing) and the stopping rules for data collection. It should identify all measures that will be collected.
• Disclosures: The manuscript should include the disclosure section required of all empirical papers at AMPPS, with placeholders for any information that is unknown prior to data collection.
• Results: Although a Stage 1 manuscript is reviewed prior to data collection, the manuscript should include a results section that conveys the analysis plan in detail. The results section should describe the full analysis pipeline, including all data preprocessing steps and any planned corrections for multiple testing. All planned analyses should be described fully, and any measures that will be used in unspecified exploratory analyses should be identified. The manuscript should include placeholders where the results will be reported, including tables and figures. AMPPS strongly encourages authors to write and test their analysis scripts in advance using simulated or pilot data to ensure that the actual results can be analyzed in the same manner. When a planned analysis is contingent on the results of another analysis, authors should write the results section with those contingencies in place (like a “choose your own adventure” book). Only those analyses described in the Stage 1 manuscript results section can be treated as planned analyses for the final, Stage 2 submission (although unplanned exploratory analyses are welcome in a Stage 2 submission; see below). The results section should fully describe any outcome-neutral criteria (e.g., definitions of ceiling/floor effects, manipulation checks) that must be met for successful testing of the stated hypotheses. Ideally, authors will submit their analysis scripts along with the Stage 1 manuscript.

Examples of Stage 1 manuscripts that have been subsequently accepted and published in AMPPS:

• Mazar et al. RRR
• Srull & Wyer RRR
• Strack RRR
• Dijksterhuis RRR
• Rand RRR
• Finkel RRR

2. The Stage 1 manuscript undergoes an initial review by the editorial team. Given our stringent criteria, some proposals will be “desk rejected” at this initial review stage (desk rejection will be based on agreement by at least two members of the editorial team).

3. Stage 1 manuscripts passing this initial screening will be sent out for expert review. Reviewers will be asked to assess:

• The importance of the research question(s)
• The logic, rationale, and plausibility of the proposed hypotheses
• The soundness and feasibility of the methodology and analysis pipeline (including statistical power)
• Whether the clarity and degree of methodological detail would be sufficient to replicate exactly the proposed experimental procedures and analysis pipeline
• Whether the authors provide a sufficiently clear and detailed description of the methods to prevent undisclosed flexibility in the experimental procedures or analysis pipeline
• Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; manipulation checks) for ensuring that the results obtained are able to test the stated hypotheses
• Whether the analysis plans are appropriate and complete and whether they make clear which analyses are tests of a priori theoretical hypothesis and which are exploratory.

4. Once the Stage 1 manuscript has been provisionally accepted, and prior to beginning their study, authors must register their approved protocol on the OSF or in another recognized repository. (OSF has a tailored mechanism for registered reports: https://osf.io/rr/.) The accepted protocol registration can be made public immediately or it can remain private until final acceptance of the Stage 2 manuscript. Following registration, authors will conduct the study as planned. They are reminded that any deviation from the stated procedures, regardless of how minor it may seem to the authors, could lead to rejection of the manuscript at Stage 2. Once the study has been completed, authors will update the Stage 1 manuscript to include the actual results and they will write an abstract and general discussion section. This complete, post-data, Stage 2 manuscript will be submitted as a revision to the provisionally accepted Stage 1 manuscript. Authors should include access to all data and materials. Authors should also provide a link to the registration for the accepted Stage 1 protocol (either a public link or an anonymized view-only link for the review process; the registration can remain private until acceptance of the Stage 2 manuscript). All registered analyses must be conducted. The paper may include additional exploratory analyses in a separate sub-section of the results description as well as in online supplementary materials. Authors must certify that all data were collected after the provisional acceptance of the Stage 1 manuscript. Apart from minor stylistic changes, the introduction may not be changed from the Stage 1 version, and the stated hypotheses may not be changed. Any textual changes (corrections of typographical errors) to the introduction or method section must be clearly marked. Any relevant literature that appeared after the provisional acceptance of the Stage 1 manuscript should be covered in the Discussion section.

5. The Stage 2 manuscript will be sent out for an expedited review process in which reviewers will be asked to assess:

• Whether the data passed the approved outcome-neutral criteria (such as absence of floor and ceiling effects or success of positive controls)
• Whether the Introduction, rationale and stated hypotheses are the same as those approved during the Stage 1 submission (required)
• Whether the authors adhered to the registered experimental procedures
• Whether any unregistered exploratory analyses are identified as exploratory, well justified, methodologically sound, and informative
• Whether the authors’ conclusions are justified given the data

6. The review process will iterate until the final manuscript is accepted. Note that the statistical significance, novelty, and conclusiveness of the findings will not factor into editorial judgments for a Stage 2 manuscript. Those were already evaluated at Stage 1, prior to data collection.

Other considerations

Withdrawn Manuscripts: Authors of a provisionally accepted Stage 1 manuscript may need to withdraw their manuscript following or during data collection. Possible reasons could include a major technical error, an inability to complete the study due to other unforeseen circumstances, or the desire to submit the results to a different journal. In all such cases, manuscripts can be withdrawn at the authors’ discretion. However, AMPPS will publicly record such withdrawals by noting the authors and title of the Stage 1 submission and a brief description of the reason(s) for withdrawal. Partial withdrawals are not possible; i.e. authors cannot publish part of a registered study by selectively withdrawing one of the planned experiments. Such cases must lead to withdrawal of the entire paper. Studies that are not completed by the agreed Stage 2 submission deadline (which can be extended in negotiation with the editorial office) will be considered withdrawn and will be subject to a Withdrawn Registration.

Incremental Registrations: Authors may add experiments to a project. In such cases, the approved Stage 2 manuscript will be accepted for publication, and authors can propose   additional experiments for Stage 1 consideration. Where these experiments extend the approved submission (as opposed to being part of new submissions), the editorial team will seek to fast-track the review process. This option may be ideal when the initial experiment reveals a major serendipitous finding that warrants follow-up within the same paper. In cases where an incremented submission is rejected (at either Stage 1 or 2), authors will retain the option of publishing the most recently approved version of the manuscript. For further advice on specific scenarios for incremental registration, authors are invited to contact the editorial office.