2025· Neural Information Processing Systems· 7 citations· 35 references
Computer Science
TL;DR
This work proposes M ulti-Model C ontrastive D ecoding (MCD), which integrates a pretrained language model with an evil model and a truthful model for contrastive decoding and effectively reduces hallucinations in LLMs and outperforms state-of-the-art methods across various benchmarks.
Abstract
Despite their outstanding performance in numerous applications, large language models (LLMs) remain prone to hallucinations, generating content inconsistent with their pretraining corpora. Currently, almost all contrastive decoding approaches alleviate hallucinations by introducing a model susceptible to hallucinations and appropriately widening the contrastive logits gap between hallucinatory tokens and target tokens. However, although existing contrastive decoding methods mitigate hallucinations, they lack enough confidence in the factual accuracy of the generated content. In this work, we propose M ulti-Model C ontrastive D ecoding (MCD), which integrates a pretrained language model with an evil model and a truthful model for contrastive decoding. Intuitively, a token is assigned a high probability only when deemed potentially hallucinatory by the evil model while being considered factual by the truthful model. This decoding strategy significantly enhances the model’s confidence in its generated responses and reduces potential hallucinations. Furthermore, we introduce a dynamic hallucination detection mechanism that facilitates token-by-token identification of hallucinations during generation and a tree-based revision mechanism to diminish hallucinations further. Extensive experimental evaluations demonstrate that our MCD strategy effectively reduces hallucinations in LLMs and outperforms state-of-the-art methods across various benchmarks.
Large Language Models (LLMs) have achieved remarkable success in natural language generation but remain prone to hallucinations—generating content that is fluent but factually incorrect. While recent inference-time interventions like Contrastive Decoding (CD) effectively mitigate this by penalizing tokens favored by a "weak" hallucination-prone model, they introduce significant computational overhead (doubling inference latency) and fail to permanently align the model. In this paper, we propose \textbf{ContrastSFT}, a novel training framework to mitigate hallucinations in LLMs that internalizes the efficacy of contrastive decoding into the model's parameters via Contrastive Logit Regularization (CLR). Unlike standard Supervised Fine-Tuning (SFT) which indiscriminately maximizes the likelihood of ground-truth tokens, ContrastSFT dynamically recalibrates the training objective by subtracting the log-probabilities of a weak reference model. This effectively penalizes "easy" but potentially hallucinatory patterns captured by the weak model, forcing the model to learn more robust, factual representations. Extensive experiments on NLU benchmarks (ParaRel, WiCE) and Factuality tasks (HaluEval, MMLU) demonstrate that ContrastSFT achieves a 5-9\% absolute improvement over SFT and previous contrastive methods. Crucially, ContrastSFT eliminates the need for auxiliary models during deployment, retaining the high inference efficiency of standard LLMs. Code will be released.
Sifan Zhou· Poster Volume 0007 The 2026...· 0 citations
SeeMe is proposed, a training-free framework that introduces the concept of feature engineering from traditional machine learning into LVLMs and restructures visual tokens through a three-stage token engineering process to suppress hallucination sources while preserving informative visual evidence.
Kai Tang, Jinhao You, Bohua Zhang et al.· 2 citations
This work studies Post Hallucination Reasoning (PHR), the stage in which hallucinated semantics enter the model's inference context and influence downstream predictions, and introduces HIVE, Hallucination Inference and Verification Engine, an evaluation infrastructure that enables controlled comparisons between faithful and hallucinated captions.
Feng He, Zhenting Wang, Qifan Wang et al.· 1 citation
This work studies hallucination from the perspective of dynamic representation shift during generation and proposes an online projection-based intervention on intermediate hidden states to suppress the hallucination-related directions, mitigating hallucinations while preserving language quality.
This work proposes CounterfactualLVLM, a training-free and plug-and-play framework that mitigates object hallucinations via small-model-assisted counterfactual reasoning and highlights the power of counterfactual guidance as a simple yet effective paradigm for enhancing factual grounding in LVLM-based multi-modal reasoning.
Xilin Li, Boyue Wang, Xiaoqian Ju et al.· Multimedia Systems· 0 citations
Fine-grained hallucination diagnosis for MLLMs is proposed, a new unified task that jointly performs hallucination detection, classification, and interpretable explanation generation and feedback experiments show that the fine-grained diagnostic explanations produced by the model effectively guide target models to correct their hallucinations.
Weilin Jin, Mingyu Wang, Wenbo Li et al.· 0 citations