FADE: Mitigating Hallucinations by Reducing Language-Prior Dominance in Large Vision-Language Models

Yichen Guo Kai Tang Fenglai Lin Yiding Sun Dongshuo Zhang Wenya Wang Linhao Cong Shanghang Zhang

Jun 2026 · arXiv.org · Vol abs/2606.29431 · 2 citations · 49 references

Computer Science

TL;DR

This work proposes FADE (FFN Attenuation for DEcoding), a training-free method that attenuates FFN outputs to reduce language-prior dominance and shows that FADE effectively mitigates hallucinations while preserving inference efficiency.

Abstract

Despite the impressive capabilities of Large Vision-Language Models (LVLMs), they remain susceptible to hallucination, generating content inconsistent with the input image. Recent studies attribute this to the dominance of language priors over visual inputs and employ contrastive decoding methods to mitigate this dominance, but the mechanistic origin remains unexplored. We investigate the information flow through each transformer layer and find that attention modules consistently aggregate visual evidence, while FFN modules at critical layers act as the source of language priors. These priors can override visual evidence, causing correct predictions in intermediate layers to drift toward incorrect outputs. Based on this insight, we propose FADE (FFN Attenuation for DEcoding), a training-free method that attenuates FFN outputs to reduce language-prior dominance. Evaluations on POPE, CHAIR, and MME benchmarks across LLaVA-1.5, mPLUG-Owl2, and InstructBLIP show that FADE effectively mitigates hallucinations while preserving inference efficiency.

DOI arXiv Semantic Scholar

View source

Similar papers

Preprint Jul 2026

SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering

SeeMe is proposed, a training-free framework that introduces the concept of feature engineering from traditional machine learning into LVLMs and restructures visual tokens through a three-stage token engineering process to suppress hallucination sources while preserving informative visual evidence.

Kai Tang, Jinhao You, Bohua Zhang et al. · 2 citations

RIVS: Mitigating Hallucination in Large Vision-Language Models via Representation Intervention on Visual Grounding Shift

This work studies hallucination from the perspective of dynamic representation shift during generation and proposes an online projection-based intervention on intermediate hidden states to suppress the hallucination-related directions, mitigating hallucinations while preserving language quality.

Xuanyu Yin, Xiaoye Qu, ∗. WeiWei · 0 citations

Conference Open access 2026

Latent Attention Denoising: A Training-Free Energy-Based Framework for Mitigating Hallucinations in Vision-Language Models

Latent Attention Denoising (LAD) is introduced, a principled and training-free framework that recasts attention calibration as a one-step score-based denoising process that achieves superior performance on both generative and discriminative tasks.

Zhiwen Luo, Siyu Jiang, Weilong Jiang et al. · Annual Meeting of the Associ... · 0 citations

Jun 2026

AURORA: Asymmetry and Update-Induced Rotation for Robust Hallucination Detection in Large Language Models

This work proposes AURORA, a novel hallucination detection framework that shifts the focus from static representations to the weight-gradient dynamics of LLMs, and achieves strong hallucination detection performance across four model families and four benchmark datasets.

Z. Zhang, Hainan Zhang, Zhiming Zheng · arXiv.org · 0 citations

Jun 2026

Vision-driven Preference Synthesis for Mitigating Hallucinations in VLMs

ViPSy (Vision-driven Preference Synthesis), a framework for constructing preference data that are both policy-aligned and visually grounded, is proposed, and experiments show that the resulting VLM, preference-aligned with ViPSy-constructed preference pairs, achieves a new state-of-the-art in hallucination mitigation.

Yunhun Nam, Jongheon Jeong · arXiv.org · 0 citations

Preprint Jul 2026

Role-Break in Attention Heads: Understanding and Detecting Hallucinations in VLMs

A lightweight linear detector is built on top of Role-Break that requires no fine-tuning of the VLM, whose feature dimension stays below 5,000 and reaches an average AUROC of 93.23 across six VLMs and four benchmarks.

Mingyu Wang, Weilin Jin, Wenbo Li et al. · 0 citations