Measure Twice, Locate Once: Mitigating Hallucinations in LLM-based Agents for Repository-Scale Fault Localization

Abstract

Fault Localization (FL) is a critical yet inherently complex phase in the software debugging process. Over the years, numerous automated FL techniques have been developed to alleviate the time and effort involved. More recently, the emergence of Large Language Models (LLMs) has marked a new era for FL. However, existing LLM-based approaches often arrive at premature conclusions due to both extrinsic and intrinsic hallucinations. To address these challenges, we propose FaultLens, a novel FL technique that equips an LLM-based agent with a fine-grained feedback mechanism for repository-scale fault localization. Specifically, the decision-making stage of our approach starts with identifying FL candidates through an LLMbased agent. Here, location extraction validation detects extrinsic hallucinations, triggering further investigation. A defined rule determines investigation completion, while a self-check mechanism mitigates intrinsic hallucinations arising from incomplete investigation. The advanced location identification stage further minimizes intrinsic hallucinations caused by faulty reasoning. We demonstrate the effectiveness of our approach through a comprehensive evaluation on the Defects4J benchmark. Our results show that FaultLens outperforms several fault localization techniques across multiple categories, including spectrum-based methods, mutation analysis, machine learning approaches, and LLM-based systems. Specifically, FaultLens achieves a 43.35% improvement over SoapFL and a 24.24% improvement over AutoFL in the Top-1 metric, surpassing state-of-the-art LLM-based agent methods. Additional experiments further indicate that FaultLens generalizes across different programming languages and LLM backends, and that its hallucination-mitigation mechanisms are transferable to an external localization workflow.