New Content From Advances in Methods and Practices in Psychological Science

Time-Related Considerations for Modeling Event-Based Data Collected via Ecological Momentary Assessment
Lizbeth Benson, Emily T. Hébert, Nicholas Hartman, et al.

Ecological momentary assessments (EMAs) and wearable devices afford opportunities to collect real-time data on events experienced in daily life. Examples of event-based data in the psychological and behavioral sciences include smoking a cigarette, experiencing a stressor, having a disruption to sleep, experiencing a depressive or manic episode, drinking an alcoholic beverage, or engaging in a bout of exercise. The increasing availability of dense sampling approaches allows for the measurement of such events at relatively fast timescales (e.g., occurring across minutes, hours, days, or weeks), expanding the possibilities for how time can be conceptualized and modeled. Survival analysis is a modeling approach that allows researchers to address scientific questions regarding whether and when events occur in time. Although not often applied to EMA data, there are myriad research questions relevant to psychosocial and behavioral scientists that can be addressed using survival analysis. In this article, we provide an overview of survival analysis, describe several time-based considerations for modeling event-based EMA data using survival analysis, and provide several illustrative examples of the different time-based considerations. Altogether, the goals of this article are to enhance knowledge of the types of research questions that can be examined using survival analysis, illustrate nuances of applying the method to EMA data, and spark ideas for future empirical and methodological research.

Using Artificial Intelligence to Generate Affective Images: Methodology and Initial Library
Maciej Behnke, Maciej Kłoskowski, Michał Klichowski, et al.

We introduce a human-in-the-loop pipeline for creating context-aware (e.g., culture, sex, and age) affect-induction images and the initial Library of AI-Generated Affective Images. Current limitations in image-based research include weak to moderate emotional-elicitation effects, limited image diversity, and minimal cultural tailoring of images. Using generative artificial intelligence (AI) guided by existing data sets and emotion taxonomies, we generated 847 images and their corresponding descriptions across 12 discrete emotions and then iteratively refined them with local cultural experts. We validated the library through six studies (N = 2,470; 58 countries). Participants rated five types of images: (a) images from existing affective databases, (b) AI-generated images without cultural adjustments, (c) AI-generated images adjusted to specific cultural contexts, (d) AI-generated images adjusted by sex (male, female), and (e) AI-generated images adjusted by age group (childhood, adulthood, older age). The AI-generated images were as effective in eliciting affective responses as the images from existing affective databases. Culturally adjusted images were slightly more effective than unadjusted counterparts in targeting intended emotions. Sex- and age-adjusted variants produced comparable responses with their base images, demonstrating controllability without loss of affective impact. Furthermore, we calculated the smallest subjectively experienced difference for affect-induction research (ds = 0.05–0.29). This work demonstrates that researchers can now generate high-quality affect-induction stimuli cost-effectively and at scale and tailor them to diverse contexts—overcoming long-standing barriers and laying the groundwork for future AI-driven methodologies in affective science.

Ensuring Transparency and Trust in Supervised-Machine-Learning Studies: A Checklist for Psychological Researchers
Hanyi Min, Feng Guo, Tianjun Sun, Mengqiao Liu, Frederick L. Oswald

Machine-learning (ML) algorithms are being rapidly incorporated into the work of psychologists given their capability and flexibility in analyzing large-scale, complex, or otherwise messy data sets. In this context and in the spirit of open science, ML research should be conducted in a transparent, understandable, and ethical manner. However, publications by psychology researchers and practitioners show a troubling lack of consistency in reporting ML information. Given that ML offers a wide range of analytical options, in this article, we address an important need by providing a comprehensive, open-science checklist that specifies the information researchers should disclose at each stage of a supervised-ML project—from data collection and preprocessing to model selection, evaluation, interpretation, and code sharing. We hope that psychological researchers will benefit from this checklist when reporting ML results and will adapt and extend this checklist further in the future.

SCORES: A Clustering Tool for Free-Text Responses
Luis Klocke, Thekla Morgenroth, Yanzhe Zeng, Benjamin Paaßen

Free-text responses are a crucial part of psychological research, enabling participants to respond without bias toward a predefined set of answers. Unfortunately, many established methods for analyzing such responses require extensive manual coding, which is time- and resource-intensive. To address this issue, automatic-processing methods based on word embeddings and clustering techniques have been proposed. In this article, we introduce SCORES (Semantic Clustering of Open Responses via Embedding Similarity), a user-friendly, graphical tool that makes such automatic methods easy to use and understand for psychological researchers.

Google-Search Data for Psychological Scientists: A Tutorial and Best Practices
Jordan W. Moon, Michael Barlev

Google searches have been described as the most important data set on the human psyche ever assembled. Google-search data—accessible through a tool called Google Trends—can provide new insights on topics as varied as stereotypes and prejudices, political attitudes, religious identity and belief, personality, motivations, psychological well-being, mental health, and culture. Google Trends can generate highly customized data sets: Users can compare the popularity of search terms across most of the world or access longitudinal data as far back as 2004, and they can do so with high geographical and temporal granularity. Notwithstanding these opportunities, Google Trends has significant limitations. Without appropriate caution, users can easily rely on data that are not meaningful or draw mistaken conclusions. We provide a comprehensive overview and tutorial covering (a) opportunities of Google Trends for psychological scientists; (b) how Google Trends scores are calculated, how reliable they are, and why some queries might yield low-quality data; (c) instructions with accompanying R code for creating custom data sets beyond what Google Trends provides by default; (d) example analyses for studies that could be done using Google Trends data; (e) an overview of common pitfalls; and (f) recommendations for safeguarding data quality and their interpretation.

Chatbots Are Undermining Crowdsourced Research in the Behavioral Sciences: Detecting Artificial Intelligence–Assisted Cheating With a Keystroke-Based Tool
Michael W. Asher, Gillian Gold, Eason Chen, Paulo F. Carvalho

Generative artificial intelligence (AI) poses a significant threat to data integrity on crowdsourcing platforms, such as Prolific, which behavioral scientists widely rely on for data collection. Large language models (LLMs) allow users to generate fluent and relevant responses to open-ended questions, which can mask inattention and compromise experimental validity. To empirically estimate the prevalence of this behavior, we analyzed keystroke data from three studies (N= 928) on Prolific between May and July 2025. Using an embedded JavaScript tool, we flagged participants who pasted text or whose keystroke count was anomalously low compared with their response length. For each flagged participant, we manually compared detected keystrokes with their final response to determine if the text could have been typed. This confirmed that despite deterrence measures, approximately 9% of participants submitted responses consistent with AI assistance or other forms of outsourced responding. These participants outperformed noncheaters (by up to 1.5SD), were more than twice as likely to share geolocations with other participants (suggesting possible proxy use), and exhibited lower internal consistency on questionnaire scales. Simulated power analyses indicate that this level of undetected cheating can diminish observed effect sizes by 10% and inflate required sample sizes by up to 30%. These findings highlight the urgent need for new detection methods, such as keystroke logging, which offers verifiable evidence of cheating that is difficult to obtain from manual review of LLM-generated text alone. As AI continues to evolve, maintaining data quality in crowdsourced research will require active monitoring, methodological adaptation, and communication between researchers and platforms.

Evaluating Cognitive Models With Permutation Testing: A Case Study of Prototype and Exemplar Categorization
Dagmar Zeithamova, Troy M. Houser, Caitlin R. Bowman

Computational cognitive models offer powerful means for testing competing theoretical frameworks. A central challenge is determining which model best explains observed data, balancing goodness of fit with parsimony. Several fruitful approaches to model comparison have been used in the areas of cognitive and mathematical psychology, but the most popular in practice remain Akaike information criterion (AIC) and Bayesian information criterion (BIC), which penalize model complexity as measured by the number of free parameters. Here, we revisit these conventional approaches to model selection on a sample case of the prototype and exemplar models of categorization. We highlight the limitations of parameter count-based complexity measures, showing that they may fail to capture a model’s true flexibility. We then introduce a Monte Carlo permutation-testing approach as an alternative that has a rich tradition in many areas but whose use for model selection is still trailing that of AIC/BIC. We demonstrate that permutation testing offers at least three advantages: more robust comparison of models with chance, more robust comparison between models with equal or differing numbers of parameters, and quantification of uncertainty in model selection. After demonstrating how permutation testing offers a more nuanced and principled framework for evaluating cognitive models, we conclude with practical considerations for implementing permutation-based model selection in cognitive-modeling research.

Just in Time or Just a Guess? Addressing Challenges in Validating Prediction Models Based on Longitudinal Data
Anna M. Langener, Nicholas C. Jacobson

A common goal of researchers using intensive longitudinal data is to develop models that predict emotions or behaviors, often using passively collected data from smartphone sensors or wearable devices. A frequent use case for such models is the development of just-in-time adaptive interventions (JITAIs). However, real-world effectiveness depends on rigorous evaluation. Previous research has highlighted challenges in selecting appropriate evaluation methods. To address these, we review key pitfalls in predictive modeling and provide recommendations for avoiding them. We focus on a common problem: the mismatch between development, evaluation and application, and use simulations to illustrate three pitfalls. First, although models may perform well from applying group-level validation (area under the curve [AUC] = .82), they may lack the ability in predicting within-persons change (mean AUC = .54,SD= .13). For JITAIs, this will prevent the model from identifying intervention-delivery moments and will discriminate only between individuals. Second, ensuring adequate variability in the outcome variable is critical. If outcomes remain stable, frequent prediction may offer little practical benefit. Third, selecting appropriate baseline models is essential; models that appear effective may underperform compared with simple baselines (e.g., AUC = .82 vs. AUC = .96). To address these pitfalls, we present recommendations for matching validation and evaluation strategies to the intended use-case scenario and provide a tool that can help researchers investigate whether their strategy and goal are misaligned. This can help improve the effectiveness of predictive models and increase their utility in real-world applications.

Realizing the Full Potential of Big-Team Behavioral Science: How Global Collaborations Can Benefit From Participatory Open-Research Practices
Netta Weinstein, Sakshi Ghai, Tia Moin, Nicole Legate, Lennia Matos, Andrew K. Przybylski

Big-team science collaborations have been heralded as a solution to oversampling in a limited number of high-income countries. Despite early successes, there is insufficient involvement from the global community and unclear benefits to globalized science. The expansion of research from sites in North America and Europe to parts of the world where most people live can create the appearance of progress based on geographical diversity while neglecting the perspectives, problems, and knowledge specific to those populations. Here, we describe participatory open-research practices that bring global perspectives to open science. Participatory practices involve revising and transparently communicating worldviews, valuing humility over control, prioritizing team facilitation over management, and listening to versus instructing collaborators. We detail these concepts and their utility and provide recommendations for conducting robust, open, and culturally embedded research that will help realize the potential value of big-team science.

Effects of Psychological Distance on Mental Abstraction: A Registered Report of Four Tests of Construal-Level Theory
Sofia Calderon, Erik Mac Giolla, Karl Ask, et al.

Construal-level theory (CLT) proposes that psychological distance influences the level of abstraction at which something is mentally construed: Things perceived as less probable (likelihood) or further away from the here (spatial distance), now (temporal distance), or self (social distance) are thought about more abstractly. In this international multilab study, we tested four basic hypotheses derived from core assumptions of CLT and explore potential moderators and boundary conditions of the effects. Participants (N= 11,775) from 27 countries and regions were randomly assigned to one of four experimental protocols focused on different types of psychological distance (temporal, spatial, social, or likelihood), and each experiment manipulated psychological distance (close vs. distant). The protocols for temporal distance (n= 2,941) and spatial distance (n= 2,973) were direct replications of Liberman and Trope (Study 1) and Fujita et al. (Study 1), respectively. The remaining two protocols were paradigmatic replications, applying to social distance (n= 2,926) and likelihood (n= 2,936). The effects of psychological distance on construal level for the four present studies were as follows (positive effects are consistent with hypotheses): temporal,d= 0.08, 95% confidence interval [CI] = [0.003, 0.16] (effect in original study:d= 0.92); spatial,d= 0.04, 95% CI = [−0.03, 0.11] (effect in original study:d= 0.55); social,d= −0.27, 95% CI = [−0.34, −0.19]; and likelihood,d= 0.03, 95% CI = [−0.05, 0.11]. Pretests indicated that valence and abstraction were confounded in response options on the outcome measure. Controlling for this confound eliminated the hypothesis-inconsistent effect of social distance,d= 0.006, 95% CI = [−0.05, 0.07]. These findings provide limited evidence for the predictions of the theory and present a critical challenge for CLT.

Fast-Track Your Abstract Screening: Mastering ASReview for Accelerating Abstract Screening and Evaluating Decisions From Automatic-Screening Methods
Tim Fütterer, Lars König, Diego G. Campos, Ronny Scherer, Steffen Zitzmann, Martin Hecht

Research syntheses, such as systematic reviews and meta-analyses, are crucial for synthesizing research to support evidence-based decision-making. However, the abstract-screening phase, during which researchers evaluate titles and abstracts for inclusion, is highly time-consuming and often results in cognitive biases and fatigue. To address these challenges, machine-learning-assisted tools, particularly those using active learning, have gained prominence. One such tool is Active Screening Review (ASReview), an open-source software for semiautomating title and abstract screening in systematic reviews. ASReview incorporates user feedback to prioritize relevant studies, reducing screening time and improving efficiency. Despite its potential, many researchers remain uncertain about integrating ASReview into their workflows and making evidence-based decisions regarding the tool’s configuration, training, and stopping criteria. In this tutorial, we provide a step-by-step guide to using ASReview, including practical examples from psychological research. We demonstrate the software’s application in two use cases: screening unlabeled abstracts using active learning and verifying results from automated-screening methods. In the tutorial, we also offer evidence-based recommendations for selecting stopping rules to balance sensitivity and efficiency. We also outline strategies for prescreening, data-set preparation, model setup, and progress monitoring to ensure that researchers can maximize the tool’s benefits while maintaining scientific rigor. By offering evidence-based guidance at each stage of the process for practitioners without coding skills, in this tutorial, we aim to help researchers harness artificial-intelligence-aided screening to enhance the quality and efficiency of research syntheses across disciplines.

Can Results-Blind Selection Improve Science Communication?
Alexa M. Tullett, Savannah C. Lewis, Nell Lambdin, Joshua Baker, Matthew Barnidge

Journalists are often maligned for covering sensational or desirable research results at the expense of studies with stronger methods. In the present study, we aimed to test how journalists’ preferences shift when studies are selected based on their methods rather than results (results-blind selection). Practicing journalists and editors, journalism faculty, and journalism graduate students (N= 413) read summaries of real social-psychology studies and rated their interest in reporting on them. Participants were randomly assigned to read either “traditional” summaries that included the results or “results-blind” summaries that excluded the results. Summaries varied on three within-subjects dimensions: replication status, preregistration status, and belief consistency. Participants expressed more interest in replicable (vs. not replicable) and preregistered (vs. nonpreregistered) studies regardless of whether they learned the results, suggesting that these studies have features that are valued by journalists. Meanwhile, results-blind selection showed potential for reducing confirmation bias, suggesting it may be worth further exploration if feasibility challenges can be addressed.

Three-Sided Testing to Establish Practical Significance: A Tutorial
Peder Mortvedt Isager, Jack Fitzgerald

Researchers may want to know whether an observed statistical relationship is either meaningfully negative, meaningfully positive, or small enough to be considered practically equivalent to zero. Such a question cannot be addressed with standard null hypothesis significance testing or standard equivalence testing. Three-sided testing (TST) is a procedure to address such questions by simultaneously testing whether a relationship is significantly bounded below, within, or above predetermined smallest effect sizes of interest. TST is a natural extension of the standard procedure of two one-sided tests (TOST) for equivalence testing. TST offers a more comprehensive decision framework than TOST with no penalty to error rates or statistical power. In this article, we give a nontechnical introduction to TST; provide commands for conducting TST in R, Jamovi, and Stata; and provide a Shiny app for easy implementation. Whenever a meaningful smallest effect size of interest can be specified, TST should be combined with null hypothesis significance testing as a standard frequentist testing procedure.

Feedback on this article? Email [email protected] or login to comment.