Item Selection as the Cause and Solution to Validity Problems in Psychology

Abstract

The validity of psychological measures depends on the selection of indicators—a process often handled subjectively and informally. We highlight problems resulting from neglected aspects of indicator sampling and investigate how item selection can help to resolve pervasive validity issues (e.g., poor content coverage, jingle-jangle fallacies). We discuss three stages of item selection: Stage 1 defines the item universe, which requires a clear, nonoverlapping delineation of psychological constructs; Stage 2 creates an initial item pool through comprehensive approaches such as cumulative item pools to ensure full content coverage; and Stage 3 selects and compiles scales using psychometric optimization procedures. We provide a synthesis of traditional and expanding selection strategies specifically designed to safeguard different aspects of validity, such as maximizing content coverage and optimizing the nomological net (convergent/discriminant validity). Overall, rigorous consideration of item selection is crucial for significantly enhancing the generalizability and validity of psychological assessments. We advocate for viewing item sampling as a strategic intervention for validity enhancement rather than a passive, ritualistic filtering step.