Watch Out for the Experienced Study Participant

This is an image of a person selecting a response on a survey.When conducting psychology studies online or in the lab, researchers might not think about participants’ past experiences as a research subject. But research published in Psychological Science suggests that these experiences could make a difference in study outcomes.

The research shows that participants who are not naïve to research methods or materials—those who have performed the same task multiple times across different studies, for example—may produce varying results, even when they do not remember having participated in similar experiments before.

Psychology researcher Jesse Chandler of the University of Michigan and Mathematica Policy Research and colleagues followed up with participants who earlier had completed a series of psychology tasks online (see Klein et al., 2014). Participants were contacted a few days, about a week, or about a month after initial participation and were asked to complete the same tasks online a second time. Some participants were assigned to the same condition they were in initially, whereas others were assigned to different conditions.

Chandler and colleagues found that the effects of the experimental manipulations were smaller in the second experiment (weighted Cohen’s d = 0.63, on average) compared to the first (d = 0.82). The decrease in effect size was greatest when participants were assigned to a different condition in the second study than they were assigned to in the first.

The authors were uncertain about the reason for this finding, writing, “There is no direct evidence of a mechanism underlying this decline.” But they speculate that there could be several potential factors at work, including practice effects, cognitive elaboration, and motivation to perform a certain way.

To ensure that study findings are as accurate as they can be, the authors suggest that that participants’ participation history should be tracked when possible and that experimenters should consider using new procedures, new materials, and larger sample sizes when conducting subsequent studies.

All data and materials from the study have been made publicly available via Open Science Framework and the article has received badges for Open Data and Open Materials.

You can learn more about some of Chandler and colleagues’ other work on collecting data online in Paolacci and Chandler (2014) and Shapiro, Chandler, and Mueller (2013).

Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26 (7), 1131-1139. DOI: 10.1177/0956797615585115

APS regularly opens certain online articles for discussion on our website. Effective February 2021, you must be a logged-in APS member to post comments. By posting a comment, you agree to our Community Guidelines and the display of your profile information, including your name and affiliation. Comments will be moderated. Any opinions, findings, conclusions, or recommendations present in article comments are those of the writers and do not necessarily reflect the views of APS or the article’s author. For more information, please see our Community Guidelines.

Please login with your APS account to comment.