Real-Time Research: How the Experience Sampling Method Is Changing Psychology

How do you design a study that captures human experience as it unfolds in real time? In this episode, Under the Cortex explores the Experience Sampling Method (ESM), a powerful approach for studying psychological processes. Host Özge Gürcanlı Fischer-Baum is joined by Jessica Fritz from Osnabruck University, and Marilyn Piccirillo from the Rutgers Addiction Research Center and Brain Health Institute, who are among the coauthors of a new article published in Advances in Methods and Practices in Psychological Science. The paper outlines ten key design and implementation considerations for ESM studies, helping researchers apply this method with clarity, rigor, and real-world relevance. 

Send us your thoughts and questions at [email protected].

Unedited Transcript

[00:00:09.780] – APS’s Özge Gürcanlı Fischer Baum 

Today on Under the Cortex, we explore how psychological research can leave the lab and meet people where they are, literally. The experience sampling method or ESM, offers researchers the ability to capture psychological phenomena as they unfold in real-time and real-life, addressing long understanding challenges around ecological validity. But there are many unknowns about how to best design an ESM study. Fortunately, a recent article in Advances in Methods and Practices in Psychological Science provides a comprehensive step-by-step guide. I am your host, Özge Gürcanlı Fischer-Baum, with the Association for Psychological Science. Joining us are Jessica Fritz from Osnabrück University and Marilyn Piccirillo from the Rutgers Addiction Research Center and Brain Health Institute. Together, they are among the co-authors of an article which provides a practical and accessible primer on how to design and implement ESM studies with concrete tips, cautionary tales, and an evolving library of resources. Jessica and Marilyn, welcome to Under the Cortex. 

[00:01:22.920] – Jessica Fritz 

Hello, Under the Cortex. Thank you so much for the invite. We are very delighted that we are here. 

[00:01:29.230] – Marilyn Piccirillo 

Hi. Thank you so much for having us. We’re really excited. 

[00:01:33.660] – APS’s Özge Gürcanlı Fischer Baum 

We are very excited to have you, too. Let me start with my first question so our listeners can learn more about you. What type of psychologist are you? Could you tell us a little bit about yourself and your academic journey. 

[00:01:48.140] – Jessica Fritz 

Yes, I’m going to start. I’m Jess. I moved from my PhD studies to the UK. I originally come from a resilience research background. I basically studied what factors help people to not develop mental health problems after an adverse event. After my PhD, I decided to move back to Germany to do my clinical training and to become a psychotherapist. That’s what I’m currently doing. I’m working as a postdoc, continuing my research. Now, my research is a bit more clinical-focused and implementation-focused, with a focus on really person-centered research. On top of my research, I’m doing my clinical training. 

[00:02:29.390] – Marilyn Piccirillo 

I I’m Marilyn. I’m a clinical psychologist. I’m also a licensed to practice. A lot of my research has focused on anxiety, trauma, and co-occurring substance use problems. I just recently wrapped up a postdoc position at the University of Washington, and then have begun a new faculty position at Rutgers Robert Wood Johnson Medical School. My area of research really focuses on how we can use smartphone data and smartphone methods to try and understand co-occurring mental health and substance use problems in everyday life, and then how we can take data, such as experience sampling data, and really use it to try and tailor or personalize clinical care. 

[00:03:10.050] – APS’s Özge Gürcanlı Fischer Baum 

Yeah, that’s great. What initially got you interested in using the experience sampling method in your studies? 

[00:03:18.880] – Jessica Fritz 

I think for me, it probably were two things. One thing that I was very much interested after studying protective factors is the component that people differ so much in clinical psychology terms, but also in any other terms. But it’s really important to take in that variability or heterogeneity between people. Particularly in clinical psychology and also in resilience research, it can be very valuable if you can build up models that are specific to a given person. And ESM is a wonderful tool that can help you with that because it helps you to collect a lot of data on a very fine-grained time scale so that you can make individual models that can capture feelings of a specific person very well and can, hopefully in the future, help us in clinical practice to also fill in some blind spots, for example, in between sessions. 

[00:04:12.330] – Marilyn Piccirillo 

Building off of what Jess said, what really speaks to me about the experience sampling method is it just allows us to understand people as they move through their everyday life. There’s all these analogies within the ESM world of looking at a video as opposed to a snapshot in time. I think that’s really It’s true. When I’m working as a clinician, I’m trying to understand how my client is practicing skills or maybe experiencing symptoms outside a session. Esm really helps us get at that. If someone’s able to take a few moments and fill out how their mood is over the course of the day, then that gives us much more insight into how things are changing for them as opposed to when they come into session and they have to fill out this retrospective measure about the past week or about the past month. Really only gives us just a snapshot, really, of what their experience has been like. 

[00:05:06.030] – APS’s Özge Gürcanlı Fischer Baum 

Yeah. Let’s talk about that a little bit, what type of individualized information this method brings. Can you elaborate on that a little more. What is special about the experience sampling method? What novel insights does it bring for psychologists? 

[00:05:24.630] – Marilyn Piccirillo 

I think what’s special about ESM is that it does allow you to capture fluctuations or changes changes because you’re sampling much more frequently. It allows you to understand how things are changing on a more granular time scale than sampling maybe every week or every month or less frequently than that. It can also… You can tailor your assessment protocols, too. You can include questions about move, but you can also include questions about context and about maybe other factors that are central to your research question and what you’re trying to study. 

[00:06:02.190] – Jessica Fritz 

Yeah, I think psychology has been criticized for a lack in ecological validity a lot because a lot of research in the past has been conducted in labs, for example, experimental research, or it has been conducted in the wild, but then in the form of, for example, cohort studies where you assess a large group of people from time to time, for example, once a year. That’s very valuable for some research questions. But when it When it comes to understanding context, to understanding fine-grained time scales and how variables change in these micro-time frames or impact each other on a very fast scale, then ESM can really be the game changer, I think. 

[00:06:50.120] – APS’s Özge Gürcanlı Fischer Baum 

We will talk about the time scales. I have a specific question about that, but I was wondering if you could tell us the difference between ESM and, let’s say, cohort approaches. What is the main difference between that? 

[00:07:06.860] – Jessica Fritz 

I think cohort approaches can be very valuable if you want to study a big group of people and understand how, in general, People change over time. Most cohort studies cover a few time points over a few years, for example, over the adolescent period, but that’s very different in ESM studies. In ESM studies, we most often focus on a smaller time frame, like on a couple of weeks or maybe a few months, and then we try to assess people as often as we can, or at least at a much higher frequency during this smaller time frame. For example, four times a day over the four weeks, or 10 times a day over seven days, or two times a day over four months. The time frame we are looking at is very different. In general, cohort studies are really good at looking at slow changes over time, like childhood to adolescence to early adulthood. And ESM studies are really good at contextualizing, at looking at daily changes or even changes within a day or changes within an hour, maybe even. So the time frame is very different, I would say. 

[00:08:20.840] – Marilyn Piccirillo 

And all of this is different than cross-sectional approaches. So cross-sectional approaches would be sampling someone at one point in time. So maybe you have a long battery of questions that are all about someone’s lifetime experience, but that’s the one time you have to really sit down and talk with them, you won’t give them those assessments again. And so that’s just a one-time snapshot. You can understand how people differ from each other. What’s unique about ESM, and I know this is tied to another question you might ask, is how you study people over time, so as compared to themselves. 

[00:08:58.960] – APS’s Özge Gürcanlı Fischer Baum 

It is providing a more fine-grained approach to their experiences. Let’s talk about some examples. In your paper, you use research topics as concrete examples to illustrate how ESM works. One of the examples that you offer is the relationship between rumination and anxiety. For this particular research example, what can an ESM approach offer to distinguish between within subjects and between subjects effects? Yeah. 

[00:09:32.370] – Marilyn Piccirillo 

I think if we start with the cross-sectional approach, that’s a traditional way of studying between subject effects. There we might sample a thousand people and ask them, Please rate your anxiety over the past month, and then please rate how often you engage in rumination. That allows us to see for people who score more highly on an anxiety measure, they also tend to report more frequent rumination. We’re comparing people to each other. But then with CSM, really, the beauty there is because you’re sampling multiple times, you can start to understand time points compared to themselves, or in other words, people compared to themselves. When someone tends to have a higher anxiety day, then they also tend to ruminate more that day. You’re looking at the comparison of time points. That’s an example of a within-person process. Now, it can get even more complex and look at between-person effects in within-person processes, but that’s more of an interaction effect. Really, the beauty of ESM is that you can study changes over time, so changes in anxiety and how those relate to changes in rumination on the day level or whatever your time scale. 

[00:10:51.660] – Jessica Fritz 

One particular aspect of ESM that Marilyn and I have been talking about in the past a lot as well is that ESM has also the advantage that you can look at N of one designs or ideographic designs or person-specific designs, case samples, however you want to call them, which basically means that you can literally take just one person and study that individual person over time because you can collect a lot of time points and then you have enough information to build an ideographic, which means a person-specific model just for that one person. That, particularly in the clinical context, can be very useful. Yeah. 

[00:11:30.030] – APS’s Özge Gürcanlı Fischer Baum 

This approach definitely gives us a lot of details about a person’s life, their experience. What about the research perspective when you are looking for responses? What are the best options for choosing and formulating responses in studies using ESM? 

[00:11:51.240] – Jessica Fritz 

Quite frankly, when we wrote down that section, there was a lot of discussion, I guess, because when we started thinking about this question, there was There’s not a lot of literature on it, or at least not specific to the ESM context. We were talking about that a lot, and I think, generally speaking, there are four different answer or response formats. One is that you can ask binary questions where people either answer yes or no. Then you can ask on a Likert scale, which means that you have a couple of response options, mostly something in between 3 and 10 response options, and then the person can choose the response option that fits best, something from never to all the time, and then all the response options in between. You have, generally speaking, provide anchors for the person, and the person can choose among these anchors. Then we have fast scales, visual analog scales. Those scales are rather often used with ESM. Those scales work in such a way that you have basically a line from zero to 100, for example. The person can indicate on a very detailed scale where he or she would locate the the answer. 

[00:13:00.980] – Jessica Fritz 

That’s much more specific than like at scales. Then the last option, and that’s a bit different from the other option because it’s not a quantitative answer, but it’s a qualitative answer, is that you can also give open-ended text boxes, for example, so that you ask a question, and then the participants can fill in their individual answer, for example, in text form. From our understanding of the literature, we think that VAS and Likert scales are the two types of scales that are most often used. Generally speaking, the VAS scale has the advantage that it’s considered continuous. Continuous data is very helpful for data analysis. Like at scale, on the other hand, has often been reported to be very user friendly because you give participants a scale. Ideally, you give anchors for all the response options. So you help the participant to give an accurate answer. And that’s also because if you give them a VAS scale, they sometimes record a 73 and they can also They’re called a 76. You cannot really be sure whether there’s a meaningful difference between 73 and 76, or at least if you would want to be sure, you would need to ask them. 

[00:14:10.100] – Jessica Fritz 

On the one hand, you give them a very fine-grained scale from zero to 100 But then on the other hand, because the scale doesn’t have that many anchors, you cannot be 100% sure what actually the participant things 73 or 76 precisely means. There are a couple of question marks that you have in your head as a researcher. But yeah, so I think it’s very difficult to answer because there are advantages and disadvantages for all these scales. For example, dichotomous scales or binary scales can be very useful if it’s a simple question that people can simply answer with a yes and a no. But then on the other hand, I think there are not as many methodological approaches that can deal with a binary scale. There are more approaches that can deal better with a Likert scale or a FAS scale. Those are all caveats that you should have in your head when choosing the response format. 

[00:15:03.060] – Marilyn Piccirillo 

I think to build off of what Jess is saying, there are these strengths and limitations of each of the different types of response scales, and it really comes down to how are you going to pair your research question and the constructs that you’re interested in studying with the items that you’re developing and the response options that go along with it, and then whatever analytic method you’re going to use to analyze and interpret that data later on. That could look different for each protocol all based on the research question and the constructs of interest. Yeah, really pretty complicated. 

[00:15:37.000] – APS’s Özge Gürcanlı Fischer Baum 

Exactly. That’s why I’m so excited that we are having this conversation so our audience can learn more about it. Let’s talk about the time scale options. What are they? What are the time scale options, what are they? What are the time scale options and why are they crucial to design ESM studies? 

[00:15:55.850] – Marilyn Piccirillo 

Time scale here refers to when you’re creating your item, over what time period are you intending to sample or intending to learn more about? Again, this gets back to what construct you’re measuring, but really also what question you’re trying to ask. You could… ESM, we’ve already decided is measuring on a more granular level, often in the course of days, hours, maybe even moments. Then do you structure your prompt to say, since the last assessment, if you’re sampling every few hours, that could be over the course of a few hours. Or maybe you say, How happy are you right now? Then that gets a sampling of how someone is doing in that moment. Or you could say, Since the last time you drink, if you’re trying to understand what’s gone on since the behavior of interest. But I think the important thing to take away here, why are they so important? Why are time skills that is so important? Is really it gets down to how much retrospection you’re asking the person to do. If it’s thinking back over the past several hours or maybe thinking back since the last time they engaged in a certain behavior, that’s going to introduce some amount of recall bias. 

[00:17:09.990] – Marilyn Piccirillo 

The longer that retrospective period is, the more opportunities there are for bias. Now, at the same time, we could go to the other end and say, Okay, well, maybe we should ask, How is everyone doing in the moment right now? That would be great. There’s no recall bias there, but that doesn’t really make sense for a lot of our psychological variables or constructs of interest. Not everyone is engaging in the behavior of interest at that moment in time. It really, again, becomes more of this strengths and limitations. There’s cost and benefits to whatever time scale you choose, and it comes down to some reflection around what level of information, what type of information are you hoping to get for that construct. 

[00:17:52.350] – Jessica Fritz 

I think related to that, what we also notice is that when we talk about time scale, we often talk about time scale schemes. It’s whether we sample data randomly or whether we have fixed times at which we sample data or whether we sample an event-based so that participants only answer the questions when a specific event has happened. But on top of that, there are other components to timescale that we should also be thinking about, not only how often do I sample, but also how long is my survey and do I need to match that with how often I can sample, and also how long is my study How can I match that with the length of my survey and the frequency of sampling? Because in the end, ESM can be burdensome because we ask a lot from patients or participants. Those are components that we should have in the back of our head when we try to figure out what’s the best time scale. I think in our experience, what can be really helpful is to do a pilot study and really test what works, what is too burdensome, what maybe has consequences for participation so that we can make the best decision with regard to time scale and a decision that also works for the participants and for compliance. 

[00:19:08.090] – APS’s Özge Gürcanlı Fischer Baum 

Because it is a fine-grained approach, I’m wondering whether it poses any issues with replicability? What are the cautionary tales about replicability and generalizability when it comes to ESM studies? 

[00:19:25.440] – Marilyn Piccirillo 

I’ll start that one. I think one of the reasons One of the reasons actually why we even wrote this paper actually comes from the panel discussion that it came from originally, which is there’s so much that’s not really known about how to design an ESM protocol because there’s all these decision points that we’ve been talking through over the past little bit now. A lot of times, those decision points never really make it into the paper. The reader is left wondering, Okay, how did they actually come up with this response option? How did they decide on that time scale? Along with our co-authors, we decided we wanted to pitch a panel and have a discussion around all of those little details that never make it into papers. From that, that’s where this paper was originated, which is great, but it does speak to the larger issue of cautionary tales. Well, there isn’t a whole lot of standardization yet because we don’t have a firm grasp on what are the standard practices for making some of these decisions. That really threatens generalizability and if we aren’t able to reproduce what other people are doing because we don’t know what the protocols themselves are. 

[00:20:38.120] – Jessica Fritz 

Yeah. I think in terms of open science, there has been a lot of development in the past years. As for all other kinds of studies, there are templates and there are resources that people can use. I think in today’s world, it’s very realistic to, for example, write preregistrations or submit protocol papers. For example, where if you don’t have a registration or protocol paper. You can also just write a thorough method section. If you have problems, for example, with the word count, it’s very often possible to have online supplements where you can put information in. There are also very nice and very helpful repositories, for example, for ESM items, where you can submit your items so that other people can reuse your items. 

[00:21:21.700] – Marilyn Piccirillo 

Really, with open science approaches, I think we collectively thought that that might be a way forward to reduce some of the threats to replicability and generalisability. If we can get people out there using what resources are available to post more about their protocols or to share their items. One of our co-authors, Olivia, currently is leading the item repository, where people can submit whatever items that they’ve put into their protocol, and then other researchers can go in and see, Okay, maybe to measure positive effect, I’ll use these items because at least I know that other people have used them as well. That’s a way of moving very slowly, but surely as a field towards more standardized or more, hopefully more generalisable practices. 

[00:22:05.920] – APS’s Özge Gürcanlı Fischer Baum 

Yeah. It is exciting to hear that this came out, this paper came out of a panel discussion where you noticed gaps in the field and said, All right, so let’s do something about that. I would like to end with that note. I would like to highlight to our listeners that your paper is a great story for researchers who want to learn about this method. They should check out the sections titled Starting Points for In-Depth Readings. They are like assignments. If you want to learn more about it, here is a list of articles that you can look at. Before we say our good vibes, is there a final key takeaway or final thought that you would like to leave with our listeners? 

[00:22:51.170] – Jessica Fritz 

Maybe we would like to thank our co-authors who have been very crucial to the conceptualization and the write-up of the paper because this was really a collective process. Only by sharing all the knowledge and the trial and error experiences of our own work, we were able to write this piece. But I don’t think that any one of us… Well, maybe some of us could have written that on our own. I definitely could not have. I’m very thankful for all the input and all the things I could learn from them. 

[00:23:22.860] – Marilyn Piccirillo 

I think to add on that, that there’s a lot of unknowns, too. One of the things that’s really encouraging to me about this field of ESM is that there’s a lot of new work coming out of researchers who are trying to learn more about these decision points and what’s the impact of making this decision or that decision. A lot of this work is under development. I think you’re not just talking about the starting points for in-depth reading, but also the little Google repository that it’s included in the paper itself to actually contribute to the ongoing updating of research being done in the field. Maybe together, not just the two of us, not just even with our panel or our co-authors, but also just collectively as a field, trying to make steps towards stronger methods in this area. But yeah. Thank you. Yeah. 

[00:24:07.120] – APS’s Özge Gürcanlı Fischer Baum 

Marilyn and Jessica, thank you one more time for joining us. 

[00:24:11.880] – Jessica Fritz 

Thank you. It was our pleasure. 

[00:24:13.410] – Marilyn Piccirillo 

Thank you for having us. 

[00:24:14.750] – APS’s Özge Gürcanlı Fischer Baum 

This is Özge Gürcanlı Fischer-Baum, with APS, and I have been speaking to Jessica Fritz from Osnabrück University and Marilyn Piccirillo from the Rutgers Addiction Resource Center and Brain Health Institute. If If you want to know more about this research, visit psychologicalscience.org. Would you like to reach us? Send us your thoughts and questions at [email protected].