Enhancing Replicability and Reproducibility in Observational-Coding Research: A Tutorial in R

Abstract

Replicating observational-coding studies is notoriously difficult: Coder variance attenuates effect sizes, protocols can be opaque, and raw text/video/audio cannot always be shared. In this tutorial, I present a three-step, open-science workflow that links design decisions to reproducibility outcomes, assuming interchangeable observers and simple mean rating scores. In Step 1, I show how (under the appropriate assumptions) the Spearman-Brown prophecy and attenuated correlation formulas translate coder reliability into sample-size targets; in an illustrative example, I illustrate that with two coders and N  ≈ 158 participants, researchers retain 80% power to detect a correlation of r = .30. In Step 2, I provide a six-step loop from training coders to calculating interrater reliability. This protocol emphasizes the importance of creating agreement matrices and running periodic interrater reliability checks; using simulated data, I demonstrate how off-diagonal errors pinpoint coder drift. In Step 3, I address coder positionality and ethical data sharing, offering a decision tree that maps media sensitivity and participant consent onto data-repository options. Annotated code, simulated data, and a brief consent template are hosted on the OSF repository for this tutorial. Adopting this pipeline enables researchers to plan, monitor, and disseminate observational work in a way that is both transparent and statistically robust.