Stimulus Selection As a Researcher Degree of Freedom: The Danger of Testing By Selecting Controlled Subsets

Using data from spoken-word recognition, we show that a commonly used methodological and statistical approach in psycholinguistics—dividing a continuous variable into “low” and “high” subsets that are matched on confounding covariates, and comparing their values on an outcome variable with a t-test—can lead to erroneous conclusions.