Automated Unit-Test Generation with GPT-5: An Empirical Comparison

Abstract

Large language models (LLMs) are increasingly proposed as automated test generators, yet small, fully reproducible measurements with current-generation models remain rare in the open literature. We report a controlled study in which four configurations of the GPT-5 model family—gpt-5-nano, gpt-5-mini, gpt-5.4, and gpt-5.4 with chain-of-thought (CoT) prompting—each generated pytest unit suites for 25 HumanEval problems drawn with a fixed seed, yielding 100 trials, 1518 executed test cases, 341 viable mutants, and 153,505 model tokens at a measured cost of $0.42. We evaluate each suite on three orthogonal axes: execution validity (does the suite import and collect tests), specification agreement (pass rate against the canonical solution), and fault-finding power (mutation kill rate of canonical-passing tests against viable syntactic mutants of the canonical solution). The compile-rate gain across the four configurations is large and, under a paired Wilcoxon signed-rank test, statistically significant for both gpt-5.4 (p=0.020) and gpt-5.4+ CoT (p=0.005) versus the nano baseline. Once a suite compiles, neither line coverage nor pass rate against the canonical solution differs significantly across configurations under the same paired test. Mutation kill rate, computed against viable regex-generated mutants of the canonical solution, improves in aggregate from 26.6% to 35.2% but per-problem paired differences are not significant in our sample. Cost rises by roughly 7x from nano to gpt-5.4+ CoT ($0.83 vs. $5.80 per 1,000 generations). All raw logs, generated tests, and analysis scripts are released for replication.