Aug 2026· ACM Transactions on Software Engineering and Methodology· 0 citations· 56 references
TL;DR
Findings indicate that cross-model collaboration offers a practical and parameter-efficient alternative to scaling up monolithic models for code generation and maintains competitive accuracy when only 20% of test cases are available for diagnostic feedback.
Abstract
Large language models (LLMs) have significantly advanced automatic code generation, yet most existing approaches rely on single-model inference, making their performance susceptible to model-specific limitations and failure patterns. To address this, we introduce Cross-Model Collaborative Scheduling (CMCS), a framework that enables multiple LLMs to collaboratively generate code. At its core, CMCS combines execution-guided self-correction with a peer-to-peer corrective handover mechanism, transferring unresolved problems between models to leverage their complementary strengths and mitigate the limitations of any single model. Experiments on eight code generation benchmarks show that CMCS achieves the strongest performance among the evaluated comparable-scale approaches with strong parameter efficiency. In particular, two 7B-scale models consistently outperform a much larger 30B-scale model on every benchmark where both are evaluated. When applied to proprietary models, CMCS boosts Pass@1 on APPS to 36.00%, more than three times that of the stronger individual model (11.33%). CMCS also maintains competitive accuracy when only 20% of test cases are available for diagnostic feedback. Ablation, scalability, and cost-effectiveness analyses quantify component contributions and practical deployment trade-offs. These findings indicate that cross-model collaboration offers a practical and parameter-efficient alternative to scaling up monolithic models for code generation.
The results indicate that functional correctness in code generation can be meaningfully improved without modifying the backbone architecture, by jointly optimizing how tasks are prompted, how the model is adapted, and how final outputs are selected.
Due to their black-box nature, LLMs suffer from limited explain- ability and a lack of determinism. Their usage cost can also rise, particularly with repetitive tasks on large codebases. To mitigate this, we conduct a novel empirical study targeting three domain- specific languages for transformation rules, namely Comby, GritQL, and Ast-Grep. We evaluate three LLMs (GPT-5.4, GPT-oss-120B, and Llama3.1-8B) on six diverse datasets covering four software- evolution tasks: API misuse correction, program repair, API migra- tion, and language version migration. Our results provide evidence that transformation rule synthesis moves beyond proof-of-concept with strong frontier models. GPT-5.4 achieves consistently high rule applicability rates and produces transformations closest to the ground truth across most benchmarks. Smaller and open-weight GPT-oss-120B and Llama3.1-8B models remain effective for simpler, localized changes but struggle with complex migration scenarios. We also observe non-negligible generalizability through the usage of meta-variables and through a high reuse score in the first quartile of many datasets. Finally, when compared to the anti-unification algorithm, LLMs outperform it in correctness, but underperform in rule applicability. Overall, our results show great potential for LLMs to generate sound, correct, generalizable, and reusable rules.
Axel Allain, Aymeric Blot, D. Khelladi et al.· 1 citation
The software engineering research community has enthusiastically embraced the integration of Large Language Models (LLMs) into complex techniques to solve a wide variety of tasks. However, the extent to which this investment is strategic remains unclear, as the native capabilities of successive frontier model generations can rapidly render existing techniques obsolete. To assess this research investment, we analyze 35 LLM-based technique papers from ICSE 2026. We evaluate whether their complex tools can be outperformed by the simplest possible alternative: a single, automatically generated prompt executed on a newer generation model, without any iterative refinement. We find that for between 37% and 63% papers, a newer model with a single prompt natively outperforms the heavily engineered tooling proposed just a year prior. We identify that constructive techniques like code generation or repair are more amenable to substitution by a single-prompt. We also identify a surviving set of papers relying on strategies that provide additional insights to the model where newer LLMs will amplify the proposed technique. Our findings raise questions about the cost-benefit proposition of techniques designed as workarounds to temporary model deficits and the need to focus on enduring challenges that scale synergistically with future model generations. Our source codes and results are made publicly available at https://github.com/less-lab-uva/What-Survives-the-Next-Model.
Nahian Salsabil, Joy Saha, S. Dristi et al.· 0 citations
Research on LLM-based code generation has achieved strong performance on function-level benchmarks, yet repository-level code generation in real-world projects still faces challenges from cross-file dependencies. Existing studies usually emphasize single-round or final-round outcomes, which limits the analysis of where feedback gains concentrate and when later repair begins to offset earlier progress. We propose a dynamic evaluation framework for repository-level code generation based on DevEval’s Python tasks. The framework constructs processed test sets, evaluates multi-round repair trajectories with unit-test feedback over multiple rounds, and characterizes trajectories via round-level effectiveness, feedback-induced gain, and regression stability. However, roughly three-quarters of the gains emerged within the first three rounds, and net feedback gain turned negative by Round 5. Previously repaired errors reappeared in later iterations, with reappeared error rates from 4.5% to 8.8% across models. These findings indicate that unit-test feedback is most effective as an early-stage signal, and that practical repair systems should adopt early-round budgeting with best-so-far retention rather than open-ended loops.
Fuze Kuang, Dongmei Liu, Yong-Li Wang et al.· International Conference on...· 0 citations
An automated, multi-dimensional evaluation framework for C# code generation, applying it to four state-of-the-art LLMs: GPT, Gemini, Claude, and Grok is presented and a substantial gap between correctness and quality attributes is revealed.
Seyed Mohammad Mahdi Ghalandarian, Majid Bazargani, Masoumeh Taromirad· 0 citations
Results provide initial evidence that KV states can serve as transferable computational representations rather than strictly model-local caches, and motivate context mobility as a systems abstraction for reducing redundant prefill across heterogeneous LLM and multi-agent inference workflows.
Yi Li, Dongming Jiang, Yi Zhao et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
