Analyzing GPT-5 performance in general chemistry

Abstract

The ability of Large Language Models (LLMs) to outperform humans on many intellectual tasks has created a difficult landscape for assessing student work. One strategy is designing assignments to be specifically difficult for AI. However, LLMs have been shown to perform well on general chemistry problems across topics, and new models continue to make AI-proofing assignments more difficult. In this work, we evaluate the recently released GPT-5 Thinking model across a set of 1736 questions from the Openstax general chemistry textbook. While GPT-5 achieves near-perfect accuracy overall, categorizing questions by the types of cognitive skills required elucidates specific weaknesses. Notably, GPT-5 tends to make major errors when asked to create figures. These findings inform best practices for AI-resistant assignment design.