Dosage Dilemma: Unpacking Meditation App Science

Although meditation apps are gaining popularity, a clear link between usage dosage and intervention outcomes has yet to be established. 

In this episode of Under the Cortex, host Özge Gürcanlı Fischer Baum teams up with Simon Goldberg from the University of Wisconsin to explore a critical question: “How does the “dosage” of meditation app use impact mental health outcomes? Drawing from a randomized controlled trial with 662 participants published in APS’s journal Clinical Psychological Science, the conversation examines various ways to measure dosage and how these metrics relate to changes in psychological distress.

Send us your thoughts and questions at [email protected].

Read related news release: Link Between Meditation App Use and Well-Being Needs Further Investigation

Unedited Transcript

[00:00:07.580] – APS’s Özge Gürcanlı Fischer Baum

As technology continues to evolve, meditation apps have surged in popularity, and scientists are now leveraging them for intervention studies. Yet the relationship between usage frequency and outcomes remains unclear. Can research help identify the optimal dosage for effective results? This is Under the Cortex. I’m Özge Gürcanlı Fischer Baum with the Association for Psychological Science. In this episode, Under the Cortex examines various approaches to defining dosage in meditation app interventions. Joining me is Simon Goldberg from the University of Wisconsin, whose recent article on this topic was published in APS’s journal, Clinical Psychological Science. Together, we will explore how meditation apps can be used most effectively for promoting mental health. Simon, thank you for joining me today. Welcome to Under the Cortex.

[00:01:01.580] – Simon Goldberg

Thank you for having me.

[00:01:03.270] – APS’s Özge Gürcanlı Fischer Baum

Could you tell us a little bit about yourself? What type of psychologist are you?

[00:01:08.310] – Simon Goldberg

I am a counseling psychologist, and my main affiliation, as you said, is at the University of Wisconsin, Department of Counseling and Psychology. I also work at a contemplative science research center called the Center for Healthy Minds here at UW.

[00:01:23.390] – APS’s Özge Gürcanlı Fischer Baum

When you are doing your counseling, you definitely got interested in studying meditation apps. What was your journey like? Why meditation apps?

[00:01:34.790] – Simon Goldberg

I actually got interested in meditation when I was in college. This was a long time ago. This was 20 years ago. There weren’t apps back then or smartphones back then. A lot of my early experience with meditation actually was from reading books. But I found these really beautiful books really helpful for learning these techniques and finding ways to understand and work with my mind. Fast forward For 15 years, obviously, technology has become a huge force, very powerful force in our society. This seems like a way that we can disseminate these techniques at scale. To some level, concern or even mistrust about the impact that mobile technology is having on our minds and on our relationships and on our societies. Disseminating meditation through these tools feels to me like a way of using this powerful force in our society for good, really trying to have technology be a beneficial force in our world.

[00:02:34.690] – APS’s Özge Gürcanlı Fischer Baum

Yeah. In your study, you explored the relationship between dosage and the interventions. How do they go together and what is an important topic to talk about?

[00:02:46.330] – Simon Goldberg

Well, I think there’s both scientific and practical reasons to be interested in dosage. From a scientific standpoint, we really don’t know as far as I can tell whether there’s a strong relationship between how much people are in this digital intervention context and how much they’re benefiting. That’s something for me as a scientist, I sit up at, and I really want to understand if there is such a relationship, and if there isn’t, why we’re not seeing that relationship from From a practical standpoint, it really makes a difference whether somebody has to practice, say 30 minutes a day to benefit from a tool like this, or whether a smaller amount of practice is equally effective. That can really impact how easy How easy it is for people to integrate this into their daily lives.

[00:03:33.530] – APS’s Özge Gürcanlı Fischer Baum

Yeah, that is definitely your contribution to the field. Your research uses various operationalizations of dosage, such as minutes of use, like you said, days of use, and types of activities completed, why did you decide to explore multiple definitions of dosage? Can you tell us a little bit more about what you just said?

[00:03:54.060] – Simon Goldberg

Yeah. In our study, like in lots of app studies, we’re able to get objective dosage information right from the app itself, which I think is really powerful, and that differentiates these technologically delivered interventions from some of the earlier generations. As we’ve been analyzing the data over the various randomized trials, that we’ve run, we’ve realized that there’s lots of different ways that we can be operationalizing dosage. We’ve just gotten curious. I’m really grateful, actually, that we’re in this age of scientific transparency, where we’re encouraged to show what we did. We don’t have to cherry-pick our findings. So in this paper, we really we embrace that by being transparent about the different ways that we looked at dosage. And conceptually, I think it really can make a difference whether we’re measuring how many activities, for example, somebody did in an app like the one we’re studying, the Healthy Minds program, or whether it’s how many days they use it or how many minutes they specifically practice meditation. Those are all correlated, of course, but they also might tap different patterns of use in potentially meaningful ways.

[00:05:03.070] – APS’s Özge Gürcanlı Fischer Baum

Yeah, conceptually, definitely, it directly addresses your research question, but I would like to highlight what you just said. It is also important to share these things openly for replication purposes, right? People can go and look at your study and then they will then know what exactly to do when they try to replicate. Let’s talk about your statistical approach to this a little bit. You also employed different strategies for modeling outcomes, including multilevel modeling and latent class analysis. How did these approaches impact the results, and why is it important to consider different modeling techniques?

[00:05:44.570] – Simon Goldberg

Yes, Also, the cheeky, maybe, subtitle that we gave it, It Depends on How You Ask, point is the short version is that the results, whether or not we saw a significant association, varied somewhat depending on how both we were operationalizing dosage, as you said before, and then also how we were modeling the outcome. Just like I was saying about the subtle differences in what dosage means, depending on how we operationalize it, the modeling approaches are also subtly different, or maybe not so subtly different. For example, some models looked at effects at post-test, whether we were seeing change from pre to post-test versus pre to follow-up, and those are different questions to some extent. Then other questions use the intensive nature of the data, the data that we collected to take advantage of the more intensive sampling. Those were the multi-level models, and then the latent class modeling actually grouped different patterns of usage into latent classes and then looked to see on a week-to-week basis how those were associated with outcomes in that week. It’s all outcome, but to us, they all felt like, in a way, defensible modeling approaches. That’s why we wanted to include them because they’re all different ways we think a reasonable scientist could analyze data like this.

[00:07:04.280] – APS’s Özge Gürcanlı Fischer Baum

Yeah, let’s talk more about that. Before, allow me to say that your methodological approach is impressive. We are talking We’re going to talk about 41 models. Here goes my question. Across the 41 models in your study, you found varying results, like you said, regarding the association between dosage and psychological distress. Could you explain why some models showed significant associations while others did not?

[00:07:33.940] – Simon Goldberg

Yeah. Explaining why feels a little tricky. I can say I think a major contributor to the variation is that some of these effects are pretty small. We had several hundred people in our study, so it’s bigger than some studies, but we didn’t have several thousand people. I think if we were running this in much larger samples, and there are data sets like that out there from these widely we used meditation apps, I think that we might be able to have more consistency across the findings. Most of the findings all went in the direction where more meditation in general was better, which is the received view, if you will, or the hypothesis that we would have ahead of time There was one exception, and there were some models where when we grouped people into dosage groupings, and specifically, we looked at people who were assigned to use the Healthy Minds program app who didn’t meditate at all or didn’t use the app at all, rather, versus those who use the app some. In some of those models, in particular at the follow-up time point, we actually see that the people who didn’t use it at all do better.

[00:08:39.680] – Simon Goldberg

That was not what we expected. That was, of all of the findings, that was the only one in a way that went in the totally opposite direction. Most of them showed at least a modest association, even if it wasn’t statistically significant, with better outcomes from more practice, if that makes sense.

[00:08:59.120] – APS’s Özge Gürcanlı Fischer Baum

Yeah. Going back to what you said, maybe people who seek meditation apps are looking for something specific, or maybe they need them. What are your thoughts about that?

[00:09:12.460] – Simon Goldberg

I think that that’s true. This was in a randomized trial context. These actually, we haven’t talked about the sample, but it was school employees here in Wisconsin during the early months of COVID. These were very high stress population. We’ve written some data showing how highly stressed they were during that time period or published some data on that. I do think that people are looking for something when they join studies and when they start using an app like this.

[00:09:38.950] – APS’s Özge Gürcanlı Fischer Baum

Following up on what you said about the direction of the results we are talking about, One of your findings suggests that higher dosage was linked to larger decreases in psychological distress, but not consistently. What factors might explain this inconsistency?

[00:09:57.310] – Simon Goldberg

As I mentioned before, I think statistical power is one of them. I think that that’s something that all of us are trying to pay attention to as much as we can. But having smaller samples, it makes it harder. In smaller, I mean less than a few thousand in some cases, if you’re detecting a relatively small effect, it can really make a difference. Another methodological issue that I want to highlight that we wrote about in the paper some is this notion of data that are missing not at random. That’s where there’s some confounding between whether data are available and what the outcome would have been had it been observed. I think in particular for the group who were randomized to use the app, didn’t use it at all, but stayed in the study, my hunch is that there are a different group than the average person who was randomized to use the app. Because I think most people who stop using an app in a trial like this are more likely to drop out of the trial. That’s what we’ve seen in other cases. I think there’s some missing not at random that can lead to these unexpected findings.

[00:11:00.440] – Simon Goldberg

And statistically, that’s really a complicated thing to deal with. There aren’t necessarily straightforward ways to just analyze that away. Another thing I want to mention that I think you alluded to some before is this notion that people are using these tools in different ways. So I think it’s maybe we started from the point of thinking of meditation or thinking of a meditation app, like we might think of psychotherapy or a medication where you take you get a higher dose, you get a higher effect. I think people are using these sorts of digital tools in much more variable, more fluid, more nuanced kinds of ways, where maybe you’re using it more, for example, when you’re not feeling well. There’s actually more engagement when your outcomes are worse in some ways, or you’re feeling better and you don’t really feel a need to be using this. You would get these unexpected relationships between dose and outcome in that case, when it’s confounded in some way with the trajectory of symptom change, where it’s not just a unidirectional relationship between dosage and outcomes.

[00:12:05.910] – APS’s Özge Gürcanlı Fischer Baum

Right. I mean, there is definitely selection bias like you talk about, and then there can be other things to know. It is a very complex question to tackle, but it is a really great starting point, your paper. Let’s talk about what else we know from the literature. How do your findings compare with previous studies on meditation apps and mental health, and particularly Really, regarding the dose-response relationship?

[00:12:33.650] – Simon Goldberg

There are other studies looking at dosage in a meditation app context out there. There aren’t a lot of them. Most of them, including work that we’ve published previously, tend to focus on one operationalization of dosage and maybe one modeling of outcome. Like I said before, we could have published that paper. We had data to show that there was a relationship or to show that there wasn’t a relationship. We could have made that decision and published a paper that way. I mean, some of the, I think, clearest data has come from really large scale data sets from publicly available apps like Calm or Headspace. There are examples in the literature, and hundreds or thousands of people where they do see relationships on the order of correlations of 0. 15, for example. So small to moderate magnitude correlations. There hasn’t been a meta-analysis that I’m aware of aggregating data from different trials. I think it’s still a little bit of an open question, but it’s something that we were really excited to look at. I’m hoping others take some inspiration from the different ways. I hope that this starts a conversation about different ways that we can be measuring and modeling the things that we did.

[00:13:50.390] – APS’s Özge Gürcanlı Fischer Baum

Yeah, I hope so, too. Let’s talk about future studies then. What are the implications of your research for future studies on mental health and apps, and particularly in terms of how researchers should operationalize and model dosage?

[00:14:08.010] – Simon Goldberg

From my perspective, I think it’s too early to say that this is the way we should be operationalizing dosage in this is the way we should be modeling outcomes. I think it’s actually worthwhile to look at it in a variety of different ways and to test multiple metrics, and I think, ideally, be transparent about that and report it. Quite frankly, I’m really grateful to APS for being able to I feel like it’s been a voice of encouraging people to be transparent in their reporting. I think a lot of journals wouldn’t have considered this paper with all of the models that we ran.

[00:14:38.180] – APS’s Özge Gürcanlı Fischer Baum

Thank you.

[00:14:39.080] – Simon Goldberg

Yeah. But more specifically, I think what we really need are studies that are manipulating dosage. Because if we’re really going to answer this in a satisfying way, we need to not just be looking in this naturalistic way. This is in a randomized trial, but we didn’t randomize people the different doses. I think that’s one piece The second piece I want to mention is the value in looking at dose effects on outcomes that are more proximal to the practice itself. Because in this trial, we did look week to week in the late class models, but most of our models focused on outcomes from pre to post, a month in between. Then our three-month follow-up was even further in time from when someone was doing a practice. That really might be too crude a level of analysis. We might actually see a lot more if we’re looking closer time to when the practice is actually happening. Things like ecological momentary assessment can be a really powerful way to look at those kinds of effects.

[00:15:37.650] – APS’s Özge Gürcanlı Fischer Baum

Yeah, no, these are great suggestions. What additional research would you recommend to better understand the relationship between this app usage and outcomes?

[00:15:47.990] – Simon Goldberg

Yeah, again, I think for me, the experimental work where we are manipulating dosage and then work where we’re looking at the more proximal effects are two of the key pieces.

[00:15:59.720] – APS’s Özge Gürcanlı Fischer Baum

What What would be your key takeaway for listeners who are interested in using meditation apps to improve their mental health?

[00:16:07.150] – Simon Goldberg

My main suggestion whenever we’re generalizing from the scientific literature to our own experience is to look at our own experience and to trust our own experience in a way, because these are all generalizations that you’re making from a population or really just a sample. For me, I’ve been meditating most days for about 20 years, and I have no doubt that the days when I meditate go better than the days when I don’t meditate, that it has a beneficial effect on my mental health and on my day. So I’m willing to spend the time every day, regardless of what the data on dosage show, because I know that from my own experience. And I think meditation is a nice thing because it’s something that people can try for themselves and see for themselves. There’s so many tools out there that are available. That’s my encouragement is to see for yourself what seems to be helpful.

[00:17:01.550] – APS’s Özge Gürcanlı Fischer Baum

Yeah. Simon, thank you so much. This was a pleasure. I personally learned a lot. I hope our listeners also enjoy our conversation.

[00:17:10.310] – Simon Goldberg

Thank you for having me.

[00:17:11.500] – APS’s Özge Gürcanlı Fischer Baum

This is Özge Gürcanlı Fischer Baum with APS, and I have been speaking to Simon Goldberg from University of Wisconsin.