CTForge is presented, an LLM-powered framework that automatically generates configuration-specific test suites and demonstrates that structured configuration-aware refinement is essential for LLM to produce effective test suites.
This paper addresses automated unit test generation with large language models (LLMs). LLM-based test generation has not yet attained a quality level sufficient for practical use in industry. Although LLMs often reproduce API syntax faithfully, they frequently disregard semantic usage constraints and execution-environment dependencies, leading to assertion failures, mock-related errors, and reference/resolution errors. A prior failure analysis of Java unit test generation using GPT-4o classified 2980 trials into eight failure patterns and identified three root-cause mechanisms: external context ignorance, internal context ignorance, and a syntax–semantics gap. Building on that analysis, this paper proposes a prompt design comprising three strategies: (1) making the execution state explicit in the generated test, (2) stating semantic constraints explicitly, and (3) injecting environment constraints prior to generation. In contrast to generic techniques such as few-shot learning or chain-of-thought prompting, each proposed strategy is tied to a specific root-cause mechanism, yielding a systematic design in which each rule is explicitly justified by its correspondence to a specific root-cause mechanism. Experiments on 298 methods with five models (GPT-4o, GPT-5, GPT-5.1-Codex, Claude Sonnet 4.5, and Gemini 2.5 Pro) show improved test execution success rates for every model, with absolute gains ranging from 1.1 to 21.1 percentage points (pp). Mock-related errors were reduced by 61.9%–99.2% relative to the baseline prompt, demonstrating effectiveness against the targeted failure patterns. Finally, conditions under which the strategies transfer to other code-generation tasks are discussed, along with limitations on their scope.
The proposed IssueExec bridges the semantic gap through domain-knowledge-enhanced test representations and filters noise via hierarchical trace analysis, which bridges the semantic gap through domain-knowledge-enhanced test representations and filters noise via hierarchical trace analysis.
Jiawei Liu, Yun Lin, Chenyan Liu et al.· arXiv.org· 0 citations
The results show that executable feedback can repair secure-code generation, but its benefits depend on the model, task, feedback entry point, and especially test coverage.
Yun-Hao Liang, Cheng-Guang Gan, Rui-Xuan Ying et al.· 1 citation
E EduPluginBench is introduced, an executable benchmark and staged admission method for generated plugins in governed software ecosystems that retains protocols, public-source provenance, raw generations, row-level decisions, audits, analysis code, and reproduction instructions.
A large empirical study on using LLMs to generate CI configurations from natural language across services and model families suggests that similarity and validity are distinct objectives for CI generation and motivate schema-aware evaluation and tooling for LLM-based configuration generation.
Results support a focused conclusion: LLM-generated review is most useful as complementary semantic guidance when paired with deployment-oriented test selection, rather than as a standalone testing artifact.
Hui-Xiang Zhen, Zhihan Zhang· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
