When Not to Answer: Confidence-Aware Response Control for Hallucination Reduction in Consumer Rights Question Answering

Abstract

Large Language Models (LLMs) are increasingly deployed in high-stakes domains such as legal assistance and consumer rights support, where response accuracy is of paramount importance; however, a critical challenge in such deployments is hallucination-the generation of factually incorrect or unverified statements that may mislead users and cause tangible harm. This work introduces the notion of a confidence-aware response control system that aims to improve the robustness of consumer rights questions answered through the use of large language models by combining RAG with multi-signal confidence assessment, which uses three different signals: semantic similarity, lexical overlap, and context coverage. These scores are then aggregated using a weighted scoring function to obtain a final confidence score, where if the confidence score is below a certain threshold value determined experimentally through grid search using F1-optimal values then the system chooses not to reply. The threshold selection process and weight configurations are empirically justified through sensitivity analysis and an ablation study. The interpretable and scalar-valued confidence indicator also offers a lightweight version of explainability, making the decision process of the system easily understandable for the operators. In an experimental setup conducted using 30 questions related to consumer rights, the system achieved 90% accuracy and hallucinations to only $\mathbf{1 0 \%}$, thus outperforming a baseline LLM (63.3% accuracy, 36.7% hallucination rate) and a RAG-only system (80% accuracy, 20% hallucination rate). This approach requires no model retraining and introduces negligible computational overhead (under 100 ms additional latency), making it practical for real-world deployment and establishing abstention-based response control as an effective safety mechanism for LLM-based legal applications.