A dataset of LLM-generated broken-to-fixed test pairs is built, where initially failing tests are repaired via a loop under zero-shot, few-shot, and chain-of-thought prompting, and the performance of prompting strategies through repair rate and post-repair coverage is compared.
Abstract
Large Language Model (LLM) pipelines can generate semantically meaningful unit tests with high coverage, but generated tests often fail to compile or execute. Many frameworks therefore use LLM-driven repair loops that iteratively re-prompt on failures until validation succeeds, but these repairs can introduce issues such as hallucinating an invented class solely to make the test pass. In this paper, we study repair-induced issues and effects of prompt strategy across three Java projects. We build a dataset of LLM-generated broken-to-fixed test pairs, where initially failing tests are repaired via a loop under zero-shot, few-shot, and chain-of-thought prompting, derive a taxonomy from 117 manually analyzed pairs (three anti-pattern categories, seven scenarios), and compare the performance of prompting strategies through repair rate and post-repair coverage.
This paper addresses automated unit test generation with large language models (LLMs). LLM-based test generation has not yet attained a quality level sufficient for practical use in industry. Although LLMs often reproduce API syntax faithfully, they frequently disregard semantic usage constraints and execution-environment dependencies, leading to assertion failures, mock-related errors, and reference/resolution errors. A prior failure analysis of Java unit test generation using GPT-4o classified 2980 trials into eight failure patterns and identified three root-cause mechanisms: external context ignorance, internal context ignorance, and a syntax–semantics gap. Building on that analysis, this paper proposes a prompt design comprising three strategies: (1) making the execution state explicit in the generated test, (2) stating semantic constraints explicitly, and (3) injecting environment constraints prior to generation. In contrast to generic techniques such as few-shot learning or chain-of-thought prompting, each proposed strategy is tied to a specific root-cause mechanism, yielding a systematic design in which each rule is explicitly justified by its correspondence to a specific root-cause mechanism. Experiments on 298 methods with five models (GPT-4o, GPT-5, GPT-5.1-Codex, Claude Sonnet 4.5, and Gemini 2.5 Pro) show improved test execution success rates for every model, with absolute gains ranging from 1.1 to 21.1 percentage points (pp). Mock-related errors were reduced by 61.9%–99.2% relative to the baseline prompt, demonstrating effectiveness against the targeted failure patterns. Finally, conditions under which the strategies transfer to other code-generation tasks are discussed, along with limitations on their scope.
This paper identifies patch verbosity as a major yet overlooked concern in LLM-based APR and proposes RECAP, a lightweight, plug-and-play adapter that attaches to existing repair frameworks after generation that achieves a substantially better size-correctness tradeoff.
Wen-Qiang Luo, J. Keung, Xiaoyu Shi et al.· 0 citations
The results suggest that organizing real-bug mechanisms as retrievable and executable feedback targets is an effective way to guide generated tests toward bug-triggering inputs and behavioral oracles.
It is shown that prompt engineering "ages" in a model-family-specific way: Newer GPT models exhibit diminishing or even negative marginal gains from structured prompting, suggesting that instruction-following and reasoning scaffolds are increasingly internalized, whereas Qwen models continue to benefit substantially from Few-Shot and CCoT.
A. Rudyk, Julian Oertel, Regina Hebig· 0 citations
Research on LLM-based code generation has achieved strong performance on function-level benchmarks, yet repository-level code generation in real-world projects still faces challenges from cross-file dependencies. Existing studies usually emphasize single-round or final-round outcomes, which limits the analysis of where feedback gains concentrate and when later repair begins to offset earlier progress. We propose a dynamic evaluation framework for repository-level code generation based on DevEval’s Python tasks. The framework constructs processed test sets, evaluates multi-round repair trajectories with unit-test feedback over multiple rounds, and characterizes trajectories via round-level effectiveness, feedback-induced gain, and regression stability. However, roughly three-quarters of the gains emerged within the first three rounds, and net feedback gain turned negative by Round 5. Previously repaired errors reappeared in later iterations, with reappeared error rates from 4.5% to 8.8% across models. These findings indicate that unit-test feedback is most effective as an early-stage signal, and that practical repair systems should adopt early-round budgeting with best-so-far retention rather than open-ended loops.
Fuze Kuang, Dongmei Liu, Yong-Li Wang et al.· International Conference on...· 0 citations
Results support a focused conclusion: LLM-generated review is most useful as complementary semantic guidance when paired with deployment-oriented test selection, rather than as a standalone testing artifact.
Hui-Xiang Zhen, Zhihan Zhang· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
