Using AI in Qualitative Analysis: Can Chatgpt and Human Raters Agree?

Washington, DC · May 2025

Poster · Methodology

Session: Poster Session III - Research Proposal Posters

Authors

Evelyn WhitePresenting Author
California State University, Sacramento
Rebecca Lucke
California State University, Sacramento
Sharn Kang
California State University, Sacramento
Taiven Haggett
California State University, Sacramento
Athena Bear
California State University, Sacramento
Jianjian Qin
California State University, Sacramento

Abstract

We propose investigating ChatGPT's potential to code qualitative responses using a human-established codebook. By comparing ChatGPT’s results to trained human raters, we aim to assess inter-rater reliability between ChatGPT and humans, and thus explore the potential of using AI to reduce the cost of qualitative analysis without compromising scientific rigor.

Other

Cookie	Duration	Description
at-rand	never	AddThis sets this cookie to track page visits, sources of traffic and share counts.
CONSENT	2 years	YouTube sets this cookie via embedded youtube-videos and registers anonymous statistical data.
uvc	1 year 27 days	Set by addthis.com to determine the usage of addthis.com service.
_ga	2 years	The _ga cookie, installed by Google Analytics, calculates visitor, session and campaign data and also keeps track of site usage for the site's analytics report. The cookie stores information anonymously and assigns a randomly generated number to recognize unique visitors.
_gat_gtag_UA_3507334_1	1 minute	Set by Google to distinguish users.
_gid	1 day	Installed by Google Analytics, _gid cookie stores information on how visitors use a website, while also creating an analytics report of the website's performance. Some of the data that are collected include the number of visitors, their source, and the pages they visit anonymously.

Cookie	Duration	Description
loc	1 year 27 days	AddThis sets this geolocation cookie to help understand the location of users who share the information.
VISITOR_INFO1_LIVE	5 months 27 days	A cookie set by YouTube to measure bandwidth that determines whether the user gets the new or old player interface.
YSC	session	YSC cookie is set by Youtube and is used to track the views of embedded videos on Youtube pages.
yt-remote-connected-devices	never	YouTube sets this cookie to store the video preferences of the user using embedded YouTube video.
yt-remote-device-id	never	YouTube sets this cookie to store the video preferences of the user using embedded YouTube video.
yt.innertube::nextId	never	This cookie, set by YouTube, registers a unique ID to store data on what videos from YouTube the user has seen.
yt.innertube::requests	never	This cookie, set by YouTube, registers a unique ID to store data on what videos from YouTube the user has seen.

Using AI in Qualitative Analysis: Can Chatgpt and Human Raters Agree?

Authors

Abstract

Cookies