A comprehensive empirical analysis of LLM-based APR techniques, focusing on how repair performance is shaped by bug complexity, fault localization, reasoning settings, and costs, reveals a nontrivial trade-off between repair effectiveness and computational cost.
Abstract
Background: Software bugs remain a critical challenge in development, necessitating effective Automated Program Repair (APR) techniques. While Large Language Model (LLM)-based APR systems have shown promise, prior studies primarily focus on overall repair effectiveness. The effects of bug complexity, fault localization, reasoning settings, and repair cost-effectiveness remain insufficiently explored. Aims: This study presents a comprehensive empirical analysis of LLM-based APR, focusing on how repair performance is shaped by bug complexity, fault localization, reasoning settings, and costs. Method: We evaluate two APR techniques (ChatRepair and CodeCorrector) using three LLMs (DeepSeek, GPT, and Llama), and examine their performance across diverse levels of bug complexity and localization strategies through a multi-dimensional empirical framework and statistical analysis. Results: Although structurally complex bugs and imprecise fault localization make repair more challenging, LLM-based APR techniques still achieve competitive repair effectiveness. Imprecise fault localization can substantially enlarge the performance gap between APR techniques. Furthermore, higher-cost LLMs and stronger reasoning settings do not consistently yield better cost-efficiency, revealing a nontrivial trade-off between repair effectiveness and computational cost. Conclusions: Over 50% of moderately complex bugs can be repaired by low-cost LLM-based APR techniques. The repair effectiveness gap between APR techniques becomes larger as fault localization becomes less precise. GPT-5 repairs 7 and 39 more complex bugs than DeepSeek-V4-pro and DeepSeek-V3.2, respectively; whereas the total repair cost of DeepSeek-V3.2 shows the best cost-efficiency performance.
The first empirical study of attention patterns in LLM-based program repair is presented, providing interpretable insights into how models process bug reports and where their attention is concentrated during repair, and indicates that stronger alignment between model attention and developer-identified key sections and phrases is associated with higher repair success.
Current research is summarized to identify key gaps and future directions to optimize LLM based APR are proposed, to assure its reliability and scalability in real world software development.
Fatmaelzahra Hamdi, Ramadam Moawad, A. Mohsen· Journal of universal compute...· 0 citations
CoFiLoc first performs structured bug report denoising to extract high-value technical information, and then progressively narrows the candidate space by integrating lightweight dynamic execution evidence, stack-trace-guided structural signals, and dual semantic-lexical ranking, before applying LLM-based reasoning over a compact set of fault-relevant methods.
Nham Cao, Nhut Tien Nguyen, Thanh Nguyen· International Conference on...· 0 citations
Large language models (LLMs) have become ubiquitous in software development, with LLM-based automated program repair tools increasingly used during code review. In this report, we explore the iterative blind use of LLMs as bug-fixers. Across multiple models and repair environments, we find that LLMs consistently claim to detect bugs in entirely bug-free programs while the rate of repair of buggy programs is less than that of the damage to correct programs. We also explore the long-term dynamics of this iterative process, and find that this frequently reaches a pseudo-bug-fixing cycle where the same changes are added and removed again ad infinitum. Lastly, via mechanistic probing, we unveil the existence of a steering vector which controls the editing propensity, suggesting that LLMs have an internal representation of ``buggy code", and that this representation is what is falsely activated to induce pseudo-bug fixing. These results provide insight towards the dynamics of fully autonomous bug-fixing systems, as well as stopping conditions under ambiguous goals.
Xietao Wang-Lin, Anton Isopoussu, Louis Mahon· 0 citations
Statically typed languages offer many advantages in software engineering, including bug prevention, enhanced code quality, and reduced maintenance costs. However, these benefits come at the expense of a steep learning curve and a slower development pace. Although known for its expressive and strong type system, Haskell often frustrates programmers as they learn and use that type system, especially when debugging type errors. We introduce Goanna, a novel type checker for Haskell that focuses on improving error diagnostics. When a type error occurs, Goanna identifies a comprehensive list of possible causes and provides a potential course of action for each. To achieve this, Goanna uses constraint logic programming and Minimal Correction Subsets (MCSes) to support its diagnostics. We evaluated Goanna’s effectiveness on 49,750 Haskell programs, demonstrating its ability to identify root causes with higher accuracy (81.6%) than traditional tools (Helium, 65.5%; GHC, 33.7%). In our benchmark study, Goanna shows performance constraints when diagnosing large programs containing complex errors, but remains responsive enough to provide real-time debugging assistance for small to medium-sized programs.
Shuai Fu, Tim Dwyer, Peter James Stuckey et al.· International Conference on...· 0 citations
CoHarden is proposed, a co-generation framework that uses the Lax signal as an in-loop convergence criterion that generates a test before any fix, then iteratively hardens the test and fix against surviving mutation patches until the generated test no longer admits Lax regressions.
Yuhao Tan, Zhibang Yang, Fangkai Yang et al.· arXiv.org· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
