Experts Struggle to Identify Human Text from AI in Study Co-Authored by UofM’s Dr. J. Elliott Casal

Sept. 7, 2023 — University of Memphis Applied Linguistics professor Dr. J. Elliott Casal recently co-authored a study that revealed trained linguists and reviewers from top academic journals were largely unsuccessful in identifying artificial intelligence (AI) versus human writing of research abstracts, with an overall positive identification rate of only 38.9%.  

Conducted along with Dr. Matt Kessler from the University of South Florida, the study is concerned with research and research writing ethics with the emergence of widely available generative artificial intelligence chatbots, such as ChatGPT.  

Specifically, the study explores the extent to which trained linguists and reviewers from top academic journals can identify whether the author of a research abstract is AI or human; the basis behind these reviewers’ decisions; and the extent to which editors of top Applied Linguistics journals believe AI tools are ethical for research purposes.   

While it is important to note that research abstracts are brief and relatively formulaic pieces of writing, the authors emphasize that they play an important role in the peer-review process and are often all that is available at no cost to potential readers. Their vulnerability to AI-generation when there is a lack of ethical consensus raises many important questions.  

The examination of human judgments, accuracy and research ethics as it pertains to AI-generated text separates this study from previous research and advances the discussion around the increasingly popular topic. The findings also showed the reviewers employed multiple rationales to judge texts, and editors were divided in the belief that there are ethical uses of AI tools for facilitating research purposes.  

“A lot of scholarly and informal discussions attend to the extent to which AI-produced texts are similar to human-produced texts, which is interesting, or they are concerned with the potential pedagogical affordances or challenges brought by this new technology,” Casal said. “However, we recognize that there are ethical grey areas in academia regarding the use of generative AI in research and research writing, or perhaps there is a lack of consensus so far, and so we wanted to adopt an exploratory ethical perspective. The findings surprised us.”  

The article produced from the study — “Can linguists distinguish between ChatGPT/AI and human writing?: A study of research ethics and academic publishing” — has gained significant traction as a discussion point among the linguistics and AI communities.   

Casal recently began his second academic year as an assistant professor of Applied Linguistics in the Department of English after completing a Postdoc at Case Western Reserve University in Cognitive Science. His research and teaching focus on corpus linguistics, disciplinary and professional discourses, second language writing pedagogy, applications of technology and natural language processing to language and writing instruction.