This paper investigates automated fault localization for verification-aware languages by comparing two paradigms: state-based and counterexample-based localization, and shows that counterexample-based approaches substantially outperform state-based localization in this setting.
Abstract
Verification-aware languages, like Dafny, integrate formal specifications directly into source code to enable static correctness checks. However, when verification fails, the feedback provided is often limited to the specific condition of the error, such as a violated postcondition, rather than the root cause of the fault. While Dafny's counterexample features provide concrete execution traces, these typically expose a single failing path per assertion failure, leaving the developer to manually look through the entire trace to locate the error. This paper investigates automated fault localization for verification-aware languages by comparing two paradigms: state-based and counterexample-based localization. Our state-based localization strategy replicates the ``snapshot''methodology of AutoFix by inferring invariants and predicates to identify suspicious program states. The counterexample-based strategy consists of a family of techniques that progressively enrich the use of verifier output: from raw counterexample extraction, to structured single-trace ranking, and to multi-trace aggregation. To validate these methods, we present an evaluation framework using MutDafny to generate a diverse mutant dataset from DafnyBench and measure localization effectiveness using the EXAM score. Our results show that counterexample-based approaches substantially outperform state-based localization in this setting. Structured ranking over a single trace yields the largest improvement over raw counterexample output, while multi-trace aggregation provides additional gains in robustness and debugging utility by increasing coverage and reducing path bias introduced by the solver. These findings demonstrate that effective fault localization in verification-aware languages depends both on using counterexample information, and how that information is structured and diversified.
Vulnerability detection via static analysis traditionally relies on security experts encoding insecure coding patterns into algorithmic rules. However, this approach often focuses on syntactic patterns and overlooks deeper semantic information in the code, such as the meanings of variable and function names. As software systems grow more complex, modeling vulnerabilities using only syntactic rules becomes increasingly challenging. In this paper, we propose a semantics-aware approach to detecting software vulnerabilities. We present SETYPE, a semantics-aware type system that can be derived directly from source code based solely on the meanings of symbols and expressions in natural language. In the SETYPE type system, both type inference and checking are performed by Large Language Models (LLMs), and a failed type check indicates a potential vulnerability. We prototype PYSETYPE to demonstrate the feasibility of SETYPE for detecting vulnerabilities in Python web applications. Our evaluation on real-world applications achieves 87% detection precision and 88% detection accuracy. Using PYSETYPE, we identified 15 potential zero-day vulnerabilities, nine of which were confirmed by developers.
Fault localization (FL) is a dominant debugging cost, yet most recent LLM-based FL systems rely on static or coverage-only signals. We introduce <sc>ANVIL-FL</sc> (Anchored Near-failure Value-Informed Localization), a two-turn, tool-free framework that combines LLM reasoning with failure-anchored runtime telemetry. A lightweight tracer runs the test session and stops at the first failure, recording project-scoped function calls/returns and exceptions. <sc>ANVIL-FL</sc> first narrows candidates by proximity to the failure chain, then ranks methods using value-informed I/O snapshots (arguments/returns) and executed code snippets, generating a Top-<inline-formula> <tex-math notation="LaTeX">$k$ </tex-math></inline-formula> list with rationales and confidence. On BugsInPy, <sc>ANVIL-FL</sc> achieves Top-1/3/5 accuracies of 57.6/68.6/70.3% on the SBFL-available subset (<inline-formula> <tex-math notation="LaTeX">$N=424$ </tex-math></inline-formula>) and 57.0/67.6/69.4% on our full in-scope set (<inline-formula> <tex-math notation="LaTeX">$N=454$ </tex-math></inline-formula>). Against the strongest SBFL baseline on the overlapping set, <sc>ANVIL-FL</sc> improves Top-1/3/5 by + 38.7/+ 39.1/+ 32.8 percentage points (pp). On an overlapping test set (<inline-formula> <tex-math notation="LaTeX">$N=452$ </tex-math></inline-formula>), <sc>ANVIL-FL</sc> approaches AutoFL’s accuracy (Top-1/3/5: 57.1/67.7/69.5% vs. 59.3/71.9/73.0%), while using about <inline-formula> <tex-math notation="LaTeX">$3\times $ </tex-math></inline-formula> fewer input and <inline-formula> <tex-math notation="LaTeX">$4.8\times $ </tex-math></inline-formula> fewer output tokens per bug. Tracing overhead is modest (median 9,394 events, 4.30 MB, 11.74 s), and the full pipeline localizes each bug end-to-end in a median of 44 s at roughly <inline-formula> <tex-math notation="LaTeX">${\$}0.008$ </tex-math></inline-formula> per bug. Ablations show that failure-chain anchoring dominates (removal: approx. −40.8 pp at Top-5), with runtime I/O and executed code providing smaller but notable gains. By combining dynamic runtime signals with structured LLM prompting, our framework demonstrates that failure-proximal runtime context can yield accurate, explainable, and cost-effective method-level fault localization within the BugsInPy setting.
Ahman J. Smith, Vijay K. Madisetti· IEEE Access· 0 citations
CoFiLoc first performs structured bug report denoising to extract high-value technical information, and then progressively narrows the candidate space by integrating lightweight dynamic execution evidence, stack-trace-guided structural signals, and dual semantic-lexical ranking, before applying LLM-based reasoning over a compact set of fault-relevant methods.
Nham Cao, Nhut Tien Nguyen, Thanh Nguyen· International Conference on...· 0 citations
This work presents CodeMechanic, a bug-property-guided system for generating constrained mit- igations for spatial memory corruption that combines a two-dimensional static and dynamic context extractor with in-prompt debugging knowledge and stepwise val- idation to limit the effect of LLM errors.
Han Zheng, Rafaila Galanopoulou, Ilia Shumailov et al.· 0 citations
LISA iteratively generates API sequences and program invariants guided by API n-gram feedback, achieving higher bug-detection rates and competitive code coverage compared with both fuzzing and prior LLM-based test generation approaches, and reporting each finding as a high-confidence bug candidate for developer confirmation.
This approach uses LLMs to infer candidate specifications solely from test code and dynamic execution traces: the LLM observes only the program interface, selected inputs, and corresponding outputs or state changes, while the implementation internals remain hidden.
Tianhai Liu, Maximilian Müller, Tobias Hey et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
