2026· Annual Meeting of the Association for Computational Linguistics· pp. 28748-28777· 0 citations· 79 references
Computer Science
TL;DR
Latent Attention Denoising (LAD) is introduced, a principled and training-free framework that recasts attention calibration as a one-step score-based denoising process that achieves superior performance on both generative and discriminative tasks.
Abstract
Visual hallucination remains a major obstacle to the reliability of Large Vision-Language Models (LVLMs). We argue that this issue originates from a fundamental statistical misspec-ification: the conventional softmax attention implicitly assumes i.i.d. noise, yet real LVLM attention patterns exhibit structured and competitive biases (e.g., attention sinks) that violate this assumption. To address this mismatch, we introduce Latent Attention Denoising (LAD), a principled and training-free framework that recasts attention calibration as a one-step score-based denoising process. LAD employs an interpretable energy function to derive an analytic score and applies a single Langevin-inspired update to actively steer corrupted attention log-its toward more faithful configurations. This intervention imposes negligible computational overhead and operates at a speed comparable to standard greedy decoding. Extensive evaluations across diverse architectures confirm that LAD achieves superior performance on both generative and discriminative tasks, effectively mitigating hallucinations while maintaining efficiency comparable to standard decoding.
This work proposes FADE (FFN Attenuation for DEcoding), a training-free method that attenuates FFN outputs to reduce language-prior dominance and shows that FADE effectively mitigates hallucinations while preserving inference efficiency.
Yichen Guo, Kai Tang, Fenglai Lin et al.· arXiv.org· 2 citations
Recent advances in large vision-language models (LVLMs) have enabled powerful multimodal reasoning by integrating visual encoders with large language models (LLMs). However, their reliability is frequently undermined by hallucinations, where generated text inaccurately describes the visual input. Although fine-tuning can mitigate this problem, it is computationally expensive and requires large, curated datasets, making training-free alternatives attractive. Among these, model editing is more promising than decoding-based approaches: decoding methods adapt outputs per input but introduce computational overhead and instability, whereas model editing modifies internal representations offline, providing a more efficient and stable solution. However, existing model-editing techniques typically rely on a single global subspace to correct hallucinations, treating all test samples identically and failing to capture diverse hallucination modes across inputs. To address this limitation, we propose a training-free hallucination mitigation framework for dynamic, per-instance suppression at test time. Our method first constructs a set of Disentangled Hallucination Subspaces, each isolating a distinct hallucination mode. During inference, the model adaptively calculates weights reflecting each input's relationship to these subspaces, guiding a dynamically combined projection that selectively suppresses the most probable hallucination directions while preserving image-grounded semantics. Extensive experiments across multiple vision-language benchmarks and LVLM families demonstrate consistent improvements, highlighting the robustness, generalizability, and efficiency of our approach.
Ali Cheraghian, Hamidreza Dastmalchi, Hamed Barzamini et al.· 0 citations
This work studies hallucination from the perspective of dynamic representation shift during generation and proposes an online projection-based intervention on intermediate hidden states to suppress the hallucination-related directions, mitigating hallucinations while preserving language quality.
: When facing real-world scenes that are densely populated with objects or contain complex occlusions, Vision-Language Models are often constrained by the language prior in autoregressive decoding, producing severe hallucination phenomena. To address this pain point that limits the reliable deployment of large models, this paper proposes Dual-branch Inference for Visual-prior Elimination, a training-free hallucination mitigation mechanism for complex scenes. By constructing a dual-branch inference structure at the inference stage and introducing a dynamic visual-confidence penalty, this mechanism effectively quantifies and suppresses the overconfidence in the content generation process, forcing the model’s output to be deeply aligned with the underlying visual features. Results on the object hallucination evaluation benchmark POPE show that, without consuming computing power for model fine-tuning, the proposed method reduces the model’s hallucination rate when objects are dense or complex occlusions exist, and brings a slight improvement in the question-answering accuracy of the model on the MSCOCO and VG datasets.
Shuguo Jiang· Journal of Electronics and I...· 0 citations
Large vision-language models (LVLMs) have demonstrated remarkable capabilities in multimodal understanding. However, they remain prone to hallucinations, generating responses that are inconsistent with the visual evidence. Existing mitigation methods largely address language-prior bias or cross-modal imbalance, while progressive visual degradation across perception and memory remains underexplored. In this work, we propose Saliency-Driven Perceptual Realignment (SDPR), a training-free framework that mitigates the degradation of visual awareness throughout inference. Specifically, we first introduce saliency-driven attention redistribution to release attention hijacked by non-semantic sink tokens, thereby recovering critical visual evidence. Second, we identify spatial distortion in the KV cache and propose saliency-driven cache alignment to preserve query-relevant visual features during generation. Finally, we introduce prior-constrained contrastive decoding to penalize unfaithful predictions induced by dominant language priors. Our proposed SDPR is robust against hallucinations due to its holistic alignment of visual awareness across the entire generative trajectory. Extensive experiments across diverse LVLM architectures show that SDPR outperforms state-of-the-art methods on both hallucination and general-purpose benchmarks, requiring no additional training and incurring minimal runtime overhead. The code is available \href{https://github.com/PengSyuChen/SDPR}{\color{blue}{here}}.
Pengxu Chen, Yao Zhu, Guangming Zhu et al.· 0 citations
TRE is a parameter-free and single-run metric that estimates hallucination risk directly from the entropy signals of a single generation, without requiring any detector training or repeated sampling, and enjoys strong generalizability, efficiency, and robustness.
Pengcheng Weng, Y. Qian, Yue Tan et al.· 0 citations