ANVIL-FL: Runtime-Aware Fault Localization for Code Generation and Debugging
TL;DR
By combining dynamic runtime signals with structured LLM prompting, this framework demonstrates that failure-proximal runtime context can yield accurate, explainable, and cost-effective method-level fault localization within the BugsInPy setting.
Abstract
Fault localization (FL) is a dominant debugging cost, yet most recent LLM-based FL systems rely on static or coverage-only signals. We introduce <sc>ANVIL-FL</sc> (Anchored Near-failure Value-Informed Localization), a two-turn, tool-free framework that combines LLM reasoning with failure-anchored runtime telemetry. A lightweight tracer runs the test session and stops at the first failure, recording project-scoped function calls/returns and exceptions. <sc>ANVIL-FL</sc> first narrows candidates by proximity to the failure chain, then ranks methods using value-informed I/O snapshots (arguments/returns) and executed code snippets, generating a Top-<inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> list with rationales and confidence. On BugsInPy, <sc>ANVIL-FL</sc> achieves Top-1/3/5 accuracies of 57.6/68.6/70.3% on the SBFL-available subset (<inline-formula> <tex-math notation="LaTeX">$N=424$ </tex-math></inline-formula>) and 57.0/67.6/69.4% on our full in-scope set (<inline-formula> <tex-math notation="LaTeX">$N=454$ </tex-math></inline-formula>). Against the strongest SBFL baseline on the overlapping set, <sc>ANVIL-FL</sc> improves Top-1/3/5 by + 38.7/+ 39.1/+ 32.8 percentage points (pp). On an overlapping test set (<inline-formula> <tex-math notation="LaTeX">$N=452$ </tex-math></inline-formula>), <sc>ANVIL-FL</sc> approaches AutoFL’s accuracy (Top-1/3/5: 57.1/67.7/69.5% vs. 59.3/71.9/73.0%), while using about <inline-formula> <tex-math notation="LaTeX">$3\times $ </tex-math></inline-formula> fewer input and <inline-formula> <tex-math notation="LaTeX">$4.8\times $ </tex-math></inline-formula> fewer output tokens per bug. Tracing overhead is modest (median 9,394 events, 4.30 MB, 11.74 s), and the full pipeline localizes each bug end-to-end in a median of 44 s at roughly <inline-formula> <tex-math notation="LaTeX">${\$}0.008$ </tex-math></inline-formula> per bug. Ablations show that failure-chain anchoring dominates (removal: approx. −40.8 pp at Top-5), with runtime I/O and executed code providing smaller but notable gains. By combining dynamic runtime signals with structured LLM prompting, our framework demonstrates that failure-proximal runtime context can yield accurate, explainable, and cost-effective method-level fault localization within the BugsInPy setting.