Ensuring Research Validity: A Checklist for Stronger Science 

Scientific credibility depends on valid research. But with growing concerns about replication failures and questionable research practices, how can scientists ensure their findings stand up to scrutiny? 

In this episode, Under the Cortex explores the VALID checklist, a newly developed tool that helps researchers systematically assess the quality of their studies. Özge Gürcanlı Fischer Baum speaks with Susanne Kerschbaumer from the University of Vienna and asks how the checklist offers tailored guidance for different types of research designs. Kerschbaumer explains how the study published in APS’s journal Advances in Methods and Practices in Psychological Science addresses this important issue. Together, they discuss why validity is more than just good methodology, how researchers can avoid common pitfalls, and what the future of scientific rigor looks like. 

You can access the checklist using the following link: www.validchecklist.com  

Send us your thoughts and questions at [email protected] 

Unedited Transcript

[00:00:06.740] – APS’s Özge Gürcanlı Fischer Baum 

How do we ensure that psychological research is truly valid, replicable, and reliable? In the wake of the replication crisis, improving research validity has become a major priority. But how can researchers systematically check for potential issues throughout their studies? This is Under the Cortex, I am Özge Gürcanlı Fischer Baum with the Association for Psychological Science. In this episode, Under the Cortex examines a new tool designed to help researchers enhance and monitor the validity of their work. Joining me is Susanne Kerschbaumer from the University of Vienna, whose recent study on this topic was published in APS’s journal, Advances in Methods and Practices in Psychological Science. We will talk about the Valid Checklist, a new tool designed by Susanne’s team to help researchers enhance and monitor the validity of their work. Susanne, thank you for joining me today. Welcome to Under the Cortex. 

[00:01:13.930] – Susanne Kerschbaumer 

Thank you for inviting me. 

[00:01:16.070] – APS’s Özge Gürcanlı Fischer Baum 

I’m going to start with our classical question. This is a question that we ask all of our guests. Could you tell us a little bit about yourself? What type of psychologist are you? 

[00:01:31.170] – Susanne Kerschbaumer 

I’m mostly working in methods and statistical analysis. This is my main research focus. I did educational psychology during my master’s, but just for the methods, to be honest. Yeah, so interesting and everything related to that in all fields of psychology. 

[00:01:52.800] – APS’s Özge Gürcanlı Fischer Baum 

Yeah, that’s great. You are really passionate about methodology, statistics. That’s fantastic. What initially What also got you interested in developing a tool to ensure researchability? 

[00:02:05.840] – Susanne Kerschbaumer 

I was trying the first steps in research during my master thesis, and I found it really hard to know what to do to make my research transparent, reliable, and valid. I was thinking, what would help me? I was thinking about the Prisma checklist, really well known, and I had contact with it before. I was thinking, Oh, I really love the format of a checklist because it’s easy to use, it helps you during practical work. This was exactly what I needed for my own research. That’s how I came up with it. Just proposed it to a supervisor. 

[00:02:47.840] – APS’s Özge Gürcanlı Fischer Baum 

Yeah. First yourself, and now you are helping to build with the tool that you discovered for your own productivity. 

[00:02:57.530] – Susanne Kerschbaumer 

I really hope so that it helps other people as Well, we read your study. 

[00:03:03.110] – APS’s Özge Gürcanlı Fischer Baum 

It is a really great idea, and it looks like a tool that everyone can use very easily. Can you tell us a little more about that? What was the process like? How did you choose what you would include in this checklist? 

[00:03:21.840] – Susanne Kerschbaumer 

At first, I was doing just a literature research, just looking for any guidance, recommendations, but also pitfalls, issues with validity to get the first list of items that could maybe make it into the checklist later. But I didn’t think it was a good idea for just me to decide what the checklist should contain. That’s why we decided to do a Delphi study with experts on research methods themselves. We found them because they published either in Psychological Methods or Advances in Methods and Practices in Psychological Science. We just e-mailed everyone that published there during the years 2021 and 2022 and asked them, Hey, are you interested in taking part in a project, creating a checklist? After I created the initial list, we did three rounds. I always just put the items into a survey and the experts rated it from one to five, how important they think this item is specifically for improving validity. This, we found the final set of items and the checklist. 

[00:04:36.410] – APS’s Özge Gürcanlı Fischer Baum 

Yeah, that’s great. We will talk about the details a little bit more, but overall, it sounds like this is a great teamwork and community work, you put the community together to develop a tool that everyone can use. That’s fantastic. How did the Delphi study contribute to the final checklist design? 

[00:04:56.660] – Susanne Kerschbaumer 

The Delphi study is the heart of the checklist, I would say. The experts decided on the content. As I said, they voted items out of the pool. They also made their own suggestions based on their experiences. Really, everything that’s in the checklist is thanks to them. I was more responsible for just creating the general form. At the beginning, I was already quite sure that I wanted to have an adaptive checklist, so not just one long list, but so you can customize it. I was providing the frame for it, and they filled it with the items they felt were important. 

[00:05:38.870] – APS’s Özge Gürcanlı Fischer Baum 

The Delphi study was a way to ask the opinions of the experts in the field. 

[00:05:45.690] – Susanne Kerschbaumer 

Yes, exactly. 

[00:05:47.440] – APS’s Özge Gürcanlı Fischer Baum 

That’s great. You used science to get opinions about science? 

[00:05:52.620] – Susanne Kerschbaumer 

Yeah, because I think it’s way better to create it if it was like 45 people were involved overall, and it’s just way better to have that many opinions and just to create the resource. 

[00:06:07.780] – APS’s Özge Gürcanlı Fischer Baum 

Yeah, that’s great. Tell us more about your vision about this checklist. How does the valid checklist help researchers improve their studyability? 

[00:06:20.210] – Susanne Kerschbaumer 

My vision is you can use it at any stage of research. I would suggest using it from the start. You just put in a tiny a bit of information about your project, and then you get your checklist and start using it during the planning phase. It helps you keep in mind the important aspects you should consider during the planning, like allocating people to groups, for example, or aspects of sampling you should take care of. Then you just work in your normal way, but you have your little guide on the side that you can use to check if you’re taking care of everything that’s necessary to maybe learn about aspects you haven’t thought before. It’s just like a little handy guide that helps you in your usual research process. 

[00:07:15.570] – APS’s Özge Gürcanlı Fischer Baum 

What are some common challenges in ensuring researchability that this tool addresses? 

[00:07:23.410] – Susanne Kerschbaumer 

What it’s based on is the assumption that people make mistakes. Maybe, for example, P-hacking or adding in covariates into models and taking them out or maybe changing their initial assumptions. Most of it is honest mistakes. How it addresses those is that it tells you about them, that you should, for example, pre-register your covariates or your exact model or how you got your sample size and not just collect until you find the result you’re looking for. It just addresses those things people maybe don’t know or forget to think about and helps them keep it in mind. Pre-register. Pre-registration is a big part of the checklist. So pre-registering every detail, every step you take and writing it all down to make it as transparent as possible. That’s just how it works. It’s made for researchers to want to improve the process. 

[00:08:28.870] – APS’s Özge Gürcanlı Fischer Baum 

Yeah. How adaptable is the checklist to different research methodologies? 

[00:08:35.840] – Susanne Kerschbaumer 

I think it is a bit adaptable. I’m sure there is more you could do, but it does adapt to really important things. One thing it takes into account is if this is a design with groups or conditions, if you’re allocating people randomly to something, if this is the case, it gives you items specifically for that. It also takes into account because this was the topic It’s really important for the experts what type of data analysis you want to do. If it’s classic inferential statistics, if you want to do Bayesian statistics, because it was really important to us to not only present one way of doing data analysis, but taking it to account multiple. You can choose that. Then you can also choose if you’re able to allocate randomly to the conditions, if you have any, and then choose if you’re testing a hypothesis or if you’re just doing exploratory research because that changes a lot. Through all those questions, you can just tick the boxes, it is adaptable to your research to give you the best recommendations. 

[00:09:51.230] – APS’s Özge Gürcanlı Fischer Baum 

This sounds like a great tool, and I’m sure there are people who already started using it. What feedback have you received from researchers who have used this tool? 

[00:10:02.020] – Susanne Kerschbaumer 

The tool has been published today. I have mostly gotten feedback from people I work with or people at the university I work at because we have introduced it to get a lot of professors, lecturers, also students at our university. Thus far, I think they were really happy with it and think it’s a great idea. The feedback we got from especially younger people, like students, students that I studied with, for example, is that it can be quite hard to use if you are a beginner or maybe not that familiar with research methods or data analysis yet. This led us to the idea of actually making a second version that is really tailored to those people, like beginners with a lot more information. But apart from that, we got really positive feedback so far. 

[00:11:03.140] – APS’s Özge Gürcanlı Fischer Baum 

That’s great. How do you see the VEAL checklist evolving in the future? 

[00:11:09.230] – Susanne Kerschbaumer 

Like I just mentioned, I think the biggest priority for me is to make an educational version. I love that it can help people learn about methods. The top priority is to make more resources. I was envisioning to have, it is on a website, it is an online tool to have a version that helps you provide little bits of information on the different checklist items, guides you a lot more. This is definitely the next step I’m going to take, develop this version with people that are familiar with teaching methods, with teaching statistics to make sure it can be even more of a resource for learning. 

[00:11:58.000] – APS’s Özge Gürcanlı Fischer Baum 

Yeah, that is a It’s really important impact for our field, right? Educating the future generation of researchers. I love that. You already mentioned that this will be online, but let’s clarify it for our listeners. Where can our listeners access the Valid checklist and learn more about its development? 

[00:12:20.510] – Susanne Kerschbaumer 

You can find it on the validchecklist.com on the internet. It’s already there. You can already access this and generate your own version of the checklist. You can also give me feedback, so my contact information is on there as well. If you have any feedback, I’m always happy to work on it and make it even better for everyone involved. As I said, it has been published today, so listeners can actually read about it starting from today. 

[00:12:51.060] – APS’s Özge Gürcanlı Fischer Baum 

Yeah, that’s great. We are recording this in February, and this podcast will be published in April. By the time they are listening to this, they can go and check everything. That’s great. Let’s go back to basics a little bit and talk about the importance of validity. What key takeaways would you like researchers to remember about study of everything. 

[00:13:16.370] – Susanne Kerschbaumer 

What I would like people to take from this is that there are so many little aspects and details, especially with big studies, complicated analysis. You have to take take care of and that it’s really important to just have a way, whatever works for you, to keep track of those things and make it transparent. Because in my personal opinion, I think being transparent is like half of it at the very least. Just reporting what you did, reporting your decisions, even if you changed something mid-study, that’s completely fine, but reporting it is the important Yeah. 

[00:14:01.150] – APS’s Özge Gürcanlı Fischer Baum 

So transparency is an important step for sure. Any other thoughts, any final thoughts or advice for our listeners who are looking to improve the rigor of their studies? 

[00:14:12.680] – Susanne Kerschbaumer 

I would definitely make use of tools like preregistration and everything that’s out there. I think it’s a combination of using all those tools that we have available and also teaching that. I think it’s really important to teach that to students to students. We help with their first research work and to do it ourselves, to be a good role model in how you should be doing research. 

[00:14:40.900] – APS’s Özge Gürcanlı Fischer Baum 

That’s a great advice. Well, Susanne, thank you so much. This was a pleasure talking to you about research methodology and your reality checklist. 

[00:14:50.950] – Susanne Kerschbaumer 

Thank you for having me. I’m really happy to talk about it and share it with everyone interested. 

[00:14:57.190] – APS’s Özge Gürcanlı Fischer Baum 

This is Özge Gürcanlı Fischer Baum with APS, and I have been speaking to Susanne Kerschbaumer from the University of Vienna. If you want to know more about this research, visit psychologicalscience.org. Would you like to reach us? Send us your thoughts and questions at underthecortex and psychologicalscience.org.