International Team Constructs Definition, Inventory of Questionable Research Practices

Psychological science has struggled to define the questionable research practices (QRPs) that can undermine the integrity of scientific findings. Now, an international team of researchers has not only proposed an overall definition of QRPs but published a comprehensive list of them—as well as ways to detect and prevent them.  

The project, published in Advances in Methods and Practices in Psychological Science, identifies and classifies 40 distinct QRPs that can skew results and inflate significance. The research team defines QRPs as “ways of producing, maintaining, sharing, analyzing, or interpreting data that are likely to produce misleading conclusions, typically in the interest of the researcher.” This definition excludes clear-cut fraud and random researcher error and instead focuses on behaviors that can be technically permissible but ethically and methodologically risky.  

“Research misconduct may grab headlines, but it’s often limited to specific cases or subfields,” said Tamás Nagy of Eötvös Loránd University who co-led the study with Jane Hergert of Humanistische Hochschule Berlin. “QRPs, by contrast, are like a death by a thousand cuts for scientific credibility—small on their own, but collectively damaging to entire disciplines. Our work is the first to define and categorize these QRPs.” 

The team used a community consensus process to define QRPs and assess their characteristics, including when they occur in the research lifecycle, their detectability, and the types of problems they cause. The collaboration began during a hackathon involving 37 researchers at the 2022 meeting of the Society for Improving Psychological Science (SIPS). Nineteen collaborators continued the project online. 

QRPs include practices such as choosing biased measurements, selective sampling, redefining group membership rules, using ad hoc covariates, hypothesizing after results are known (HARKing), and publishing studies selectively, the researchers pointed out.  

They mapped each QRP to one of six research phases—planning, data collection, processing, analysis, writing, and publication—and grouped them under conceptual umbrella terms to capture clusters of related behaviors. For example, the “cherry-picking” umbrella encompasses a range of data-selection strategies that distort results, while the “p-hacking” umbrella includes 13 different manipulative statistical practices.  

“One of our key contributions is showing how broad terms like p-hacking can mask the underlying diversity of questionable practices,” Nagy said. “Disentangling these behaviors helps make specific QRPs easier to identify and address.” 

The researchers also cataloged harms, ranging from inflated effect sizes and reduced replicability to biased error rates and compromised generalizability. 

In their paper, “Bestiary of Questionable Researcher Practices,” the authors emphasized the importance of open science practices in preventing, detecting, and ultimately reducing the occurrence of QRPs.  

“While open science practices typically don’t prevent QRPs outright, they often make them easier to detect,” Nagy said. 

The paper offers several strategies to prevent risky research practices and flawed conclusions, including  

• transparent reporting of data exclusions, manipulations, sample-size decisions, measured variables, conditions, and outcomes; 
• preregistration of hypotheses, methods, and analysis plans before data collection; 
• blind data analysis to reduce confirmation bias and analytical flexibility; 
• use of best practices, standardized protocols, validated measures, and appropriate statistical models; 
• robustness checks, including sensitivity analyses and assumption checks to ensure conclusions don’t depend on a narrow set of analytic choices; 
• employment of reliable and valid instruments for measurement; and 
• public sharing of supplementary information such as data and appendices. 

The authors said they hope their work will guide researchers, reviewers, and institutions toward more rigorous and transparent scientific practices. 

Feedback on this article? Email [email protected] or login to comment.

Reference

Nagy T, Hergert J, Elsherif MM, et al. (2025) Bestiary of Questionable Research Practices in Psychology. Advances in Methods and Practices in Psychological Science, 8(3).