Log in
Back to #small language model
#small language model Open access

Training-free counterfactual hallucination mitigation method for large vision-language models

Xilin Li Boyue Wang Xiaoqian Ju Yongli Hu Wentong Wang
Aug 2026 · Multimedia Systems · Vol 32 · 0 citations · 48 references

TL;DR

This work proposes CounterfactualLVLM, a training-free and plug-and-play framework that mitigates object hallucinations via small-model-assisted counterfactual reasoning and highlights the power of counterfactual guidance as a simple yet effective paradigm for enhancing factual grounding in LVLM-based multi-modal reasoning.

Abstract

Despite the remarkable progress enabled by large vision-language models (LVLMs) in visual question answering, these models remain vulnerable to object hallucination—a critical issue where generated answers reflect plausible but nonexistent visual elements. This phenomenon stems from LVLMs’ overreliance on linguistic priors and training-set biases, undermining their trustworthiness in real-world applications. To tackle this challenge, we propose CounterfactualLVLM, a training-free and plug-and-play framework that mitigates object hallucinations via small-model-assisted counterfactual reasoning. The core idea is to explicitly contrast how LVLMs behave when visually salient objects are deliberately removed. To achieve this, we employ a debiased auxiliary VQA model to locate the most semantically influential objects in an image for a given question. These key objects are masked to generate counterfactual samples, which simulate targeted visual uncertainty. We then introduce a visual contrastive decoding mechanism that subtracts the hallucination-prone output distribution (from counterfactual input) from the original one, thereby suppressing hallucinated predictions without any fine-tuning or parameter modification. Extensive experiments across four challenging benchmarks (POPE, VQA-CP v1, VQA-CP v2, and OK-VQA) and two representative LVLMs (LLaVA and InstructBLIP) show that CounterfactualLVLM consistently reduces object hallucination tendencies and improves VQA accuracy, yielding modest but consistent gains without updating the target LVLM. Our work highlights the power of counterfactual guidance as a simple yet effective paradigm for enhancing factual grounding in LVLM-based multi-modal reasoning.

DOI Semantic Scholar
Read PDF

Similar papers

RIVS: Mitigating Hallucination in Large Vision-Language Models via Representation Intervention on Visual Grounding Shift

This work studies hallucination from the perspective of dynamic representation shift during generation and proposes an online projection-based intervention on intermediate hidden states to suppress the hallucination-related directions, mitigating hallucinations while preserving language quality.

Xuanyu Yin, Xiaoye Qu, ∗. WeiWei · 0 citations
Conference 2026

ContrastSFT: Contrastive Logit Regularization Supervised Fine-Tuning for Mitigating Hallucinations in Large Language Models

Large Language Models (LLMs) have achieved remarkable success in natural language generation but remain prone to hallucinations—generating content that is fluent but factually incorrect. While recent inference-time interventions like Contrastive Decoding (CD) effectively mitigate this by penalizing tokens favored by a "weak" hallucination-prone model, they introduce significant computational overhead (doubling inference latency) and fail to permanently align the model. In this paper, we propose \textbf{ContrastSFT}, a novel training framework to mitigate hallucinations in LLMs that internalizes the efficacy of contrastive decoding into the model's parameters via Contrastive Logit Regularization (CLR). Unlike standard Supervised Fine-Tuning (SFT) which indiscriminately maximizes the likelihood of ground-truth tokens, ContrastSFT dynamically recalibrates the training objective by subtracting the log-probabilities of a weak reference model. This effectively penalizes "easy" but potentially hallucinatory patterns captured by the weak model, forcing the model to learn more robust, factual representations. Extensive experiments on NLU benchmarks (ParaRel, WiCE) and Factuality tasks (HaluEval, MMLU) demonstrate that ContrastSFT achieves a 5-9\% absolute improvement over SFT and previous contrastive methods. Crucially, ContrastSFT eliminates the need for auxiliary models during deployment, retaining the high inference efficiency of standard LLMs. Code will be released.

Sifan Zhou · 0 citations
Preprint Jul 2026

SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering

SeeMe is proposed, a training-free framework that introduces the concept of feature engineering from traditional machine learning into LVLMs and restructures visual tokens through a three-stage token engineering process to suppress hallucination sources while preserving informative visual evidence.

Kai Tang, Jinhao You, Bohua Zhang et al. · 2 citations
Preprint Aug 2026

TruthLens: Object Hallucination Detection via Self-Evaluating Truthfulness Scores in LVLMs

TruthLens is a self-evaluation framework that teaches the LM head to expose a per-object truthfulness signal without any auxiliary model or additional inference cost, and generalizes effectively to benchmarks with substantially larger label spaces.

Yanqi Wu, Runhe Lai, Xinhua Lu et al. · 0 citations
Preprint Jul 2026

Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction

Verifier-Guided Decoding (VGD), a decoding framework in which a lightweight verifier examines each emerging object mention, rolls back the KV cache when the mention is identified as high risk, suppresses the object and its synonyms, and regenerates the affected continuation, achieves state-of-the-art object hallucination reduction.

Lei Yang, Xinze Liu, Dayan Wu et al. · 0 citations

Related blog posts