Tests increasingly participate in the decisions made by large language models and software engineering agents. They specify intended behavior, guide program construction and repair, select candidates, constrain transformations, and provide execution evidence for software analysis. These uses draw on test-driven development, yet differ substantially in test order, oracle availability, editable artifacts, and the role of execution. We present a structured scoping survey organized around the question of what decision a test changes. The review integrates 87 research and supporting records, with method- or protocol-level extraction for 83 records, alongside a separate collection of five practice resources. We distinguish the Red--Green--Refactor cycle from test-conditioned generation, execution-guided refinement, test-mediated analysis, and evaluation-only testing. We then compare code generation, repair, translation, refactoring, clone detection, code search, localization, training-data construction, and formal-specification validation. A dedicated analysis examines how agent workflows and reusable skills encode testing procedures and how their effects are evaluated. Across these tasks, the evidence supports treating test availability, test validity, feedback use, and evaluation independence as separate properties. Test passing alone does not establish behavioral equivalence, effective feedback, or process adherence; aggregate improvements can also conceal different outcomes across models, tasks, and denominators. We synthesize these distinctions into a mechanism taxonomy, a cross-task comparison, and a protocol-sensitive evidence analysis, and identify research directions in oracle validation, causal evaluation, long-horizon maintenance, and reusable test-driven agent capabilities
Yun-Hao Liang, Cheng-Guang Gan, Rui-Xuan Ying et al.· 0 citations
Generating tests from a natural-language specification requires both an input that exposes faulty behavior and a correct expected output. These requirements need not improve together: a model can increase test correctness by choosing easier inputs, or discover useful inputs whose expected outputs it cannot predict. We study this interaction through executable reward decomposition and suite-level oracle-conversion measurement. Our generator jointly emits five input--output tests in one response. During training, audited reference programs provide correctness feedback, while a fixed bank of faulty programs provides two utility signals: potential input kill and effective kill after checking the generated output. An additive GRPO objective preserves both signals without requiring execution at inference time. On an audited TC-Bench split with 506 training and 142 evaluation tasks, three independently trained Qwen3.5-9B runs at step 75 increase full-test correctness from 28.59\% to 42.54\%, input kill from 24.06\% to 25.27\%, and effective full kill from 12.23\% to 14.15\%. Matched 50-step ablations reveal a trade-off: removing kill rewards yields higher correctness and slightly higher full kill, but lowers input kill to 21.60\%. A fixed-input source--oracle crossover on 64 training-pool tasks attributes the principal NoKill-to-FullKill difference to harder input selection rather than worse output prediction on identical inputs. These results identify oracle conversion as a measurable bottleneck and show the benefits and limits of preserving input-utility feedback in joint test generation.
Yun-Hao Liang, Cheng-Guang Gan, Rui-Xuan Ying et al.· 0 citations
This work studies test utilization through matched prompting controls, paired semantic interventions, and test suites selected by fault detection to make test-specified rule changes measurable alongside implementation capability and benchmark correctness.
Yun-Hao Liang, Cheng-Guang Gan, Rui-Xuan Ying et al.· 1 citation
Public tests are widely used to guide large language model code generation, but whether models treat them as executable specifications or merely as extra prompt context remains unclear. We study test-driven code generation on HumanEval+, MBPP+, and recent LiveCodeBench tasks using Qwen2.5-Coder-7B and Qwen3.6-27B. We compare natural-language-only prompts with relevant visible tests, shuffled outputs, irrelevant tests, assertion-only tests, and stronger-model-generated synthetic tests. Evaluation combines hidden or private test pass rates with task-level behavior flips, linear probes, and layer-wise hidden-state shifts. Visible tests substantially improve Qwen2.5 performance on MBPP+ but have little or unstable effect on HumanEval+ and LiveCodeBench. For Qwen3.6 on LiveCodeBench, the natural-language-only pass rate rises from 13.1% to 39.4%, yet relevant original tests add only 2.9 percentage points (p = .458), and synthetic high5 tests add 1.7 points over their matched baseline (p = .701). More tests produce larger representation shifts, but accuracy gains quickly saturate or disappear; assertion-only and irrelevant-test prompts can strongly alter hidden states without improving correctness. After controlling for prompt length, task-level shifts are associated with both beneficial and harmful flips, with a substantially stronger association for harms. These results show that tests influence code models through both semantic guidance and prompt-context perturbation, and that representational change alone does not demonstrate effective test utilization.
Yunhao Liang, Chengguang Gan, Ruixuan Ying et al.· 0 citations
The results show that executable feedback can repair secure-code generation, but its benefits depend on the model, task, feedback entry point, and especially test coverage.
Yun-Hao Liang, Cheng-Guang Gan, Rui-Xuan Ying et al.· 1 citation
An audit-and-placebo protocol is proposed that separates verifier artifacts, interaction scaffolding, and grounded feedback credit in evaluations of self-evolving test generators in evaluations of self-evolving test generators.
Yunhao Liang, Chengguang Gan, Ruixuan Ying et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
