Improving Bug Detection in LLM-Generated Unit Tests: Revisiting Test-Oracle Reliability Across Modern Large Language Models

Abstract

The widespread adoption of Large Language Models (LLMs) has given rise to a new generation of automated unit-test generation tools, such as Codium CoverAgent, CoverUp, and GitHub Copilot, promising to reduce the manual burden of writing tests while preserving bug-detection quality. However, a recent study revealed a fundamental issue: these tools may generate tests that are effectively "designed to pass" rather than designed to catch bugs, because the test oracle is frequently derived from the very code under test rather than from an independent specification of correct behavior. This paper presents a Replication and Extension Study that re-examines this issue using a newer generation of large language models (e.g., Claude Sonnet 4.6, GPT-5, and open-source models such as Qwen3-Coder), using the Refactory dataset, which contains 1,783 real buggy Python programs written by actual university students alongside 2,442 correct reference solutions. This paper presents a formal mathematical model for categorizing the outcome of generated-tests into four classes, a couple of basic metrics: Bug-Revealing Rate (BRR) and Bug-Validating Rate (BVR); and two basic statistical tests: McNemar test and Wilson confidence interval to ensure that the results are rigorous. The paper sets out to address a direct research question: has the effectiveness of bug detection increased with the capabilities of LLMs, or does it stem from a structural problem that is not dependent on the underlying model? The Bug-Revealing Rate increases from 22.9% (GPT-4) to 39.1% (Qwen3-Coder) for all four models, reflecting an improvement in model ability to reveal bugs; however, the ability of the strongest model (Qwen3-Coder) to validate bugs decreases from 41.1% to 26.2%, suggesting that the test-oracle design problem is significantly reduced but not eliminated by model strength.