DEHALL is proposed, an automated end-to-end unit test generation tool that systematically mitigates both types of hallucinations through comprehensive context construction and targeted static analysis-based repair.
Abstract
Unit testing plays a critical role in ensuring software quality and reliability in large-scale industrial environments. While Large Language Models (LLMs) offer promising automated test generation capabilities, their practical deployment faces significant challenges due to hallucination problems. In this paper, we analyze compilation failures from LLM-generated unit tests in Ant Group's production systems and identify two fundamental types of hallucinations: extrinsic hallucinations caused by insufficient contextual information and intrinsic hallucinations stemming from model limitations even with adequate context. To address these issues, we propose DEHALL, an automated end-to-end unit test generation tool that systematically mitigates both types of hallucinations through comprehensive context construction and targeted static analysis-based repair. Our approach builds a heterogeneous graph to capture relevant context and employs specialized repair mechanisms for import, field, and method issues. Evaluation on Ant Group's internal datasets reveals that DEHALL achieves 71.56% line coverage and 67.18% branch coverage, significantly outperforming vanilla LLM approaches. In the public benchmarks, it also shows better performance on coverage and better defect detection capability than previous state-of-the-art approaches. DEHALL has been successfully deployed across multiple business domains at Ant Group, achieving an 81% developer adoption rate with positive user feedback on productivity improvements.
Large language models (LLMs) have significantly advanced automated program repair (APR), yet existing evaluations remain largely result-centric and provide limited insight into hallucination during repair. In APR, hallucination may arise not only in final patches but also in the intermediate artifacts that guide patch generation. To address this gap, we perform a multi-layered analysis of hallucination throughout the APR process. Specifically, we characterize hallucination as the production of patches or intermediate artifacts that are not faithfully grounded in the available repair evidence. We examine repair hallucination in final patches and understanding hallucination in intermediate artifacts through three tasks, namely triggering testcase identification, line coverage prediction, and additional testcase generation.We then evaluate three representative LLMs on 832 Defects4J bugs through automatic evaluation and manual analysis. Our results show that both repair and understanding hallucinations remain prevalent. Across models and settings, only 21.0%-55.9% of generated patches pass the developer-written test suite. Moreover, although more accurate intermediate artifacts are generally associated with successful repairs, this relationship does not always hold. Manual analysis of 812 sampled repairs identifies repair hallucinations in 72.7% of cases, including patches that pass all available tests; incorrect causal localization and incorrect repair strategies account for 45.9% and 18.5% of these hallucinations, respectively. Meanwhile, models frequently misidentify triggering testcases, mispredict line coverage involving branching control flow, and generate additional testcases with missing bug-triggering conditions or incorrect expected behavior.
Xuemeng Cai, Jiakun Liu, Linhan Yang et al.· 0 citations
EAHC is proposed, an Execution-Anchored Hallucination Calibration reranking framework that anchors reasoning judgments to execution behavior so that execution-equivalent candidates receive consistent scores, which implements a dual-channel architecture.
Guang Yang, Xing Hu, Xiang Chen et al.· 0 citations
The results recast package hallucination as both a measurement problem and a decoding-time control problem, and they demonstrate that the choice of defense must be matched to the threat model and recommendation utility.
Albérick Euraste Djiré, Iyiola E. Olatunji, Melissa Tessa et al.· 1 citation
The results suggest that no single architecture guarantees factual reliability, however, contextual grounding and verification mechanisms can significantly improve response quality and highlight the importance of combining language modelling capabilities with grounding strategies to support the development of more reliable AI systems.
Fault Localization (FL) is a critical yet inherently complex phase in the software debugging process. Over the years, numerous automated FL techniques have been developed to alleviate the time and effort involved. More recently, the emergence of Large Language Models (LLMs) has marked a new era for FL. However, existing LLM-based approaches often arrive at premature conclusions due to both extrinsic and intrinsic hallucinations. To address these challenges, we propose FaultLens, a novel FL technique that equips an LLM-based agent with a fine-grained feedback mechanism for repository-scale fault localization. Specifically, the decision-making stage of our approach starts with identifying FL candidates through an LLMbased agent. Here, location extraction validation detects extrinsic hallucinations, triggering further investigation. A defined rule determines investigation completion, while a self-check mechanism mitigates intrinsic hallucinations arising from incomplete investigation. The advanced location identification stage further minimizes intrinsic hallucinations caused by faulty reasoning. We demonstrate the effectiveness of our approach through a comprehensive evaluation on the Defects4J benchmark. Our results show that FaultLens outperforms several fault localization techniques across multiple categories, including spectrum-based methods, mutation analysis, machine learning approaches, and LLM-based systems. Specifically, FaultLens achieves a 43.35% improvement over SoapFL and a 24.24% improvement over AutoFL in the Top-1 metric, surpassing state-of-the-art LLM-based agent methods. Additional experiments further indicate that FaultLens generalizes across different programming languages and LLM backends, and that its hallucination-mitigation mechanisms are transferable to an external localization workflow.
Feiyu Chen, Guowei Yang, Cheryl Lee et al.· ACM Transactions on Software...· 0 citations
This work introduces UniHall, a fine-grained dataset grounded in a unified taxonomy spanning Object, Instruction, and Knowledge dimensions, and proposes Self-Adaptive Multimodal Fuzzing (SAMF), a self-adaptive framework that employs evolutionary mutation strategies to explore the boundaries of model hallucinations.
P. Zhou, Jiajun Song, Zhiwei Tang et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
