A Protocol to Identify Large Language Model Use in Undergraduate Chemistry Essays

Abstract

Large language models (LLMs) such as ChatGPT have been widely adopted by chemistry undergraduate university students as a learning tool, but few methods exist to measure the scope of their influence on essay writing. This report introduces a protocol based on monitoring the frequency of specific words used excessively by LLMs among a sample of 1073 chemistry student essays between 2018 and 2025. Analysis revealed that 54 keywords known to be favored by LLMs simultaneously increased in frequency among essays submitted during the last two years. When the lists of keywords were applied to specific essays, the findings indicated 43% of essays submitted during 2025 had at least some evidence of LLM-generated text. This protocol offers a simple method to generate an approximate scope of LLM uptake among a cohort, which can be combined with student survey data to produce a detailed overall impression of students’ use of generative artificial intelligence. This work’s findings are intended to stimulate discussion on the best practices to update literature review assessment criteria, to ensure it is maintained as a valid assessment option within a curriculum equipped for the widespread adoption of LLMs.