The Biggest Threat to Online Data Collection Is Humans, Not Bots

Researcher practices, not platforms • Humans, not bots • An emerging threat • Focusing on methodology and controls

Though online surveys have made collecting data easy and efficient, the results sometimes spark concern. A 2020 online study by the Centers for Disease Control and Prevention claimed Americans were drinking bleach to prevent COVID-19 infection, for example, but attempts at replication revealed significant issues with data quality and integrity (Gharpure, 2020; Litman, 2023).

“There’s a lot of problematic data that’s skewing our very important research,” said Leib Litman, professor at Lander College, cofounder of CloudResearch, and author of one of the attempted replication studies.

“Naturally the question becomes, ‘Okay, where’s all this bad data coming from?’” he continued. “There was a bit of a mass hysteria among researchers.”

In a 2022 Perspectives in Psychological Science paper, Webb and Tangney blamed bots. They reported that less than 3% of questionnaire responses in their sample came from humans; the rest of the responses were deemed to be poor quality and as such, from bots. However, recent papers dispute these findings, saying most of the poor-quality responses are from humans and that the onus is on the researchers to do more careful analysis and accept the time and money costs of quality research.

Still, as Justin Sulik, a postdoc at Ludwig Maximilian University of Munich and author of one of the recent papers said, “You can’t automatically trust the data you get online.”

Now, as fears of bots transform into fears of artificial intelligence (AI) taking over research studies, psychological scientists are determined to figure out the best practices for online surveys, who—or what—is behind bad data, and how to best protect surveys from the new and emerging threat of AI.

Researcher practices, not platforms

Sulik thought the fear of bots was overblown. Although data collection online is not perfect, the finding that 97% of responses were from bots seemed unrealistic to him.

“Depending on the platform, anywhere between 5% to 10% of the data are likely to be junk. And that’s just something you have to embrace,” he said. “We know that the kinds of failure rates that the paper we were responding to was reporting are just not feasible.”

Rather than doing basic quality checks prior to working with the data, like those embedded in other platforms, Webb and Tangney reported on raw data from the Amazon Mechanical Turk (MTurk) platform, “which I couldn’t recommend anyone ever do,” Sulik said. “That’s like having a drink from a river that is fed directly from a sewer outflow pipe and then getting surprised if you suddenly have stomach troubles.”

MTurk is an online platform that connects researchers with participants, allowing easy recruitment and data collection. It has fallen out of favor in the last few years as other platforms, such as Prolific and CloudResearch, offer better quality controls.

Besides using raw data, Webb and Tangney deemed respondents as participants even if they didn’t complete the study and also assumed failed attention checks meant that the respondents were bots. In a 2024 Perspectives in Psychological Science paper, Sulik took issue with these methods and assumptions. It is typically considered best practice to exclude incomplete questionnaires, and humans are known to fail attention checks at a similar rate as reported in the paper. Even when people are online for entertainment—like watching YouTube or Netflix—they have attention lapses, so it’s not a surprise to see the same in online surveys.

“It’s very easy for you to have a momentary lapse of attention despite trying really hard,” he said. “It doesn’t mean that, overall, your data are poor quality.”

Sulik also wrote that the researchers didn’t pay the participants enough for such a long and boring task, so participants may have lacked motivation and rushed through it. Not everyone is going to read every instruction and consent form carefully, so it’s important for researchers to set it up in a way that makes the participant’s job easier.

“If you had gone through this survey from the perspective of the participant or just asked a friend who can be honest with you to do it, they will tell you that this was frustrating and boring,” Sulik said. “If you solve that problem, you’ve made the other behavioral data-quality issue just a lot easier.”

Sulik said there still may be a small percentage of responses that are bots, but that the bigger problem is careless humans.

“I’m frankly more concerned about those,” he said.

Humans, not bots

Litman had a similar suspicion.

“Looking into this now for more than 10 years, it is just very clear that there is a big data-quality problem online, but it’s not bots, it’s humans,” he said.

In a 2026 Perspectives on Psychological Science paper, Litman detailed a series of studies performed between 2018 and 2023 to determine the nature and effects of low-quality data and understand who was behind the problematic accounts.

Surveys targeting MTurk respondents with a history of high- or low-quality data revealed the latter respondents were likely humans in India. When shown pictures of objects that have different names in American English and Indian English (such as eggplant/brinjal), 96% of the problematic respondents gave the Indian-English names.

Responses to other questions were nonrandom and suggested human cognition. For example, respondents answered yes more often than no (as people may do to qualify for more studies) and showed priming effects known to impact humans (such as guessing a lower height for Mount Everest after being given a lower number as an arbitrary reference point).

Next, Litman and his team performed video interviews with problematic respondents.

“We wanted to really verify that these were people,” he said. “Whenever we saw a survey that sort of looked like a bot, we just invited the participant to a video interview and then we were able to actually see them and talk to them.”

Litman found evidence of a global industry created to make money from filling out surveys. He found social networks where people exchange ideas and information on how to infiltrate surveys and get paid for them. He also uncovered Facebook posts where people buy and sell MTurk worker identification numbers.

“Very conclusively and very compellingly we showed that what looks like a bot is actually just a person who’s in a different country, who doesn’t speak English very well or at all, and ultimately their goal is to make money with a survey,” Litman said.

An emerging threat

But now a string of recent papers has turned attention to a new potential problem for online data collection: artificial intelligence.

Sean Westwood, a government professor at Dartmouth College, published a paper in 2025 demonstrating how an AI agent he created responded to survey questions indistinguishably from a human and evaded detection methods.

“When I started this, it was mostly theoretical. I was just wondering if it was possible for an AI to provide coherent responses to survey questions because bots have been a problem on surveys for as long as online surveys have existed, but they’ve usually been fairly basic,” he said. “As a part of this project, it became very clear though that AI can now perfectly mimic a human being and escape basically all of the traps that have been created to try and implement quality control standards on surveys.”

Westwood acknowledged that people rushing through surveys to get their financial reward is still probably the most common form of fraud, but malicious actors selling accounts overseas could have sophisticated tools that would be almost impossible to detect. He said that if someone like him, a government professor, can build an AI agent like this, then larger groups trying to turn a profit can certainly build these tools.

“It is possible to detect AI if you’ve built in really complicated code to track that in a survey, but it’s hard to do that,” Westwood said. “It’s only going to capture AI use in very narrow situations. Outside of that, it’s probably practically impossible.”

But Sulik and Litman disagreed with Westwood’s assessment of AI’s involvement.

“The paper was claiming that it is possible to circumvent a lot of the current state-of-the-art filters, but just because it’s possible doesn’t mean that it’s common,” Sulik said.

Litman urged people to read Westwood’s paper carefully.

“There’s a big difference between having your own agent take your own survey versus having your own agent intercept a survey that’s being fielded on a platform,” he said. “It’s like the difference between taking money out of a piggy bank and taking money out of a vault in a bank.”

“At CloudResearch, for example, we have technology that can identify AI agents with pretty much 100% accuracy,” he continued.

Whenever someone fills out a survey, Litman’s team looks at behavior metrics: how whoever is responding moves their mouse, how long it takes to answer a question, where they click.

“As of now, there are no AI agents whose behavior metrics look like humans’,” he said.

But Westwood was able to create an AI to evade these: It simulates mouse movements, key presses, hesitations, and typos, effectively mimicking humans.

Moreover, a preprint from other researchers revealed that while verified humans failed AI-detection tests around 2% of the time, respondents to surveys on Prolific, CloudResearch, and MTurk failed 6%, 10%, and 40%, respectively (Zhang et al., 2026).

Westwood doesn’t have an answer to what can be done to stop AI interference. “It’s not impossible to continue to do high-quality work. It just might mean that we have to go back in time” to using student samples and address-based sampling, he said.

He emphasized he’s not trying to kill online surveys, but he is trying to remain cognizant of where the technology is and do work he believes in.

Litman agreed that people are worried, but they haven’t yet abandoned online research.

“I haven’t seen people go back to the horse-and-buggy days of data collection, but people are concerned, and they’re looking for solutions at this point,” he said.

Focusing on methodology and controls

For now, Litman and Sulik said the focus should be on how the research is done and what controls can be put in place to prevent future problems.

“I think for careless people, which as far as I’m concerned is the bigger problem, you really need a strategy that is well thought out, in that it applies both to your audience or sample and to your particular research needs,” Sulik said.

Instead of responding to the latest panic by writing off entire platforms, “People should really take it as an opportunity to improve how they’re doing science,” he said.

“Online research is its whole specific skillset,” he continued. “People tend to think, ‘I just design a survey like I normally would, I put it online, done.’ But pivoting from in lab or in person to online really does demand a lot of know-how, skills, thought, consideration, and reflection.”

Litman wrote a free online book, Research in the Cloud: An Introduction to Modern Methods in Behavioral Science, with chapters devoted to techniques to identify and remove problematic respondents.

He said it comes down to two things: technology and methodology. Choosing a platform that was built in response to the bot crisis eliminates a lot of these data-quality issues. Furthermore, there are techniques to help weed out problematic participants. For example, adding questions that ask about impossible scenarios (such as eating at a fictional restaurant) can detect people that answer yes to everything, while open-ended follow-up questions can ensure people are reading carefully and understanding the questions.

Technology and better methodology can also help researchers get ahead of the AI problem.

“It’s not the threat at the moment that it’s made out to be in some current publications, but within a short time it very well could be, so people should start to incorporate solutions,” Litman said. “Let’s not wait for some nefarious actors to develop these things, lets develop these things ourselves so that we know how to deal with that threat internally.”

Westwood stressed the importance of working together.

“I think it’s important that we have conversations about data quality as a field, as social scientists,” he said. “I hope that we recognize that this is a threat to us as researchers and try to come up with ways to surmount and adapt.”

But as of now, Litman said, “It’s still the case that the greatest threat to online data quality is humans.”

Back to top

Feedback on this article? Email [email protected] or login to comment.