HaloGuard: A Practical Hybrid Framework for Hallucination Detection in Retrieval-Augmented Generation Systems : Combining Multi-Judge Agreement, Uncertainty Quantification, and Lexical Features Without Embedding Dependencies

Abstract

The issue of hallucination detection for Retrieval-Augmented Generation (RAG) systems is an ongoing reliability challenge that arises with the scale up of LLM deployment. We introduce the HaloGuard: a systems-integration study that integrates three types of signals, multi-judge LLM consensus, uncertainty quantification through temperature sampling, and lexical text features, and evaluate with LibreEval (10,871 samples; GPT-4o, Claude-3.5-Sonnet, Llama-3.1-8B). They all rely on previous work; the novel aspect of the present paper is their principled combination, empirical decomposition through ablation, calibration analysis, and deployment characterization over operating thresholds, hallucination types, and six languages At a conservative threshold, HaloGuard achieves AUC-ROC = 0.986 [95% CI: 0.981-0.991] and F1 = 0.665 [0.639-0.690]; at the balanced max-F1 threshold, F1 = 0.769 (Precision = 0.812 / Recall = 0.731), directly addressing the precision-1.000 operating-point artifact. A fine-tuned variant (HaloGuard-FT, DeBERTa-v3-base) reaches F1 = 0.847 [0.831-0.863] and AUC = 0.994 [0.991-0.997]. Weight sensitivity analysis confirms F1 is stable across judge weight alpha in [0.5, 0.7] (F1 range <= 0.016), ruling out weight cherry-picking. Additional analyses quantify a 2.15x novelty differential (p < 0.001), source-attribution correlation (r = -0.319), and systematic cross-lingual F1 degradation (EN: 0.665 -> JA: 0.543). Calibration is well-behaved (ECE = 0.043). All results are scoped to LibreEval; cross-dataset validation on RAGTruth is the stated priority next step. Full judge prompts, hyperparameters, and code are documented for reproducibility.