The results suggest that organizing real-bug mechanisms as retrievable and executable feedback targets is an effective way to guide generated tests toward bug-triggering inputs and behavioral oracles.
Abstract
Large language models (LLMs) have opened new opportunities for unit test generation, but executable tests do not necessarily reveal real defects. This paper studies how historical real-bug mechanisms can be transformed into executable feedback targets for LLM-based unit test generation. The proposed framework constructs structural and semantic representations of real-bug records, retrieves mechanisms applicable to a focal method, and instantiates them as synthetic bugs that guide iterative test enhancement. We evaluate the approach on method-level real-bug detection tasks from Defects4J and show that mechanism-guided synthetic-bug feedback improves real-bug detection over execution-, coverage-, mutation-, knowledge-, and search-based baselines. The results suggest that organizing real-bug mechanisms as retrievable and executable feedback targets is an effective way to guide generated tests toward bug-triggering inputs and behavioral oracles.
This paper presents a formal mathematical model for categorizing the outcome of generated-tests into four classes, a couple of basic metrics: Bug-Revealing Rate (BRR) and Bug-Validating Rate (BVR); and two basic statistical tests to ensure that the results are rigorous.
Zeyad Farooq Lutfi· Al-Noor Journal of Engineeri...· 0 citations
LISA iteratively generates API sequences and program invariants guided by API n-gram feedback, achieving higher bug-detection rates and competitive code coverage compared with both fuzzing and prior LLM-based test generation approaches, and reporting each finding as a high-confidence bug candidate for developer confirmation.
Results support a focused conclusion: LLM-generated review is most useful as complementary semantic guidance when paired with deployment-oriented test selection, rather than as a standalone testing artifact.
Hui-Xiang Zhen, Zhihan Zhang· International Conference on...· 0 citations
This paper addresses automated unit test generation with large language models (LLMs). LLM-based test generation has not yet attained a quality level sufficient for practical use in industry. Although LLMs often reproduce API syntax faithfully, they frequently disregard semantic usage constraints and execution-environment dependencies, leading to assertion failures, mock-related errors, and reference/resolution errors. A prior failure analysis of Java unit test generation using GPT-4o classified 2980 trials into eight failure patterns and identified three root-cause mechanisms: external context ignorance, internal context ignorance, and a syntax–semantics gap. Building on that analysis, this paper proposes a prompt design comprising three strategies: (1) making the execution state explicit in the generated test, (2) stating semantic constraints explicitly, and (3) injecting environment constraints prior to generation. In contrast to generic techniques such as few-shot learning or chain-of-thought prompting, each proposed strategy is tied to a specific root-cause mechanism, yielding a systematic design in which each rule is explicitly justified by its correspondence to a specific root-cause mechanism. Experiments on 298 methods with five models (GPT-4o, GPT-5, GPT-5.1-Codex, Claude Sonnet 4.5, and Gemini 2.5 Pro) show improved test execution success rates for every model, with absolute gains ranging from 1.1 to 21.1 percentage points (pp). Mock-related errors were reduced by 61.9%–99.2% relative to the baseline prompt, demonstrating effectiveness against the targeted failure patterns. Finally, conditions under which the strategies transfer to other code-generation tasks are discussed, along with limitations on their scope.
No detectable linear relationship exists between requirement properties and oracle accuracy in this dataset, suggesting that pretraining coverage and the semantic specificity of the required behavior dominate oracle correctness.
Tiancheng Ma, Nasir U. Eisty· arXiv.org· 0 citations
CoHarden is proposed, a co-generation framework that uses the Lax signal as an in-loop convergence criterion that generates a test before any fix, then iteratively hardens the test and fix against surviving mutation patches until the generated test no longer admits Lax regressions.
Yuhao Tan, Zhibang Yang, Fangkai Yang et al.· arXiv.org· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
