Object Hallucination in large vision-language models (LVLMs), where models generate non-factual content about input images, remains a critical barrier to their reliability in real-world applications. Existing mitigation strategies can be categorized into training-based and training-free methods. Training-based methods often achieve strong performance but are costly, requiring extensive computational resources, large-scale data, and time-consuming fine-tuning. Training-free approaches are particularly appealing due to their efficiency. However, existing training-free methods either require multiple decoding rounds, which adds computational overhead, or modify internal states in a model-specific way that risks degrading pretrained knowledge. We propose Test-Time Hallucination Mitigation (TTH) method, a novel training-free method that addresses both limitations. TTH introduces a token-validator module, implemented as a zero-shot Multi-Modal Classifier (MMC), to generate auxiliary logits grounded in the input image. These logits are fused with the original LVLM outputs at the token level for object tokens selected from a candidate pool. An entropy-based weighting scheme is then applied to enable robust and accurate predictions. Extensive experiments across multiple LVLM families and diverse benchmarks demonstrate that TTH consistently improves accuracy and robustness, underscoring its generalizability and practical effectiveness. Code is released at https://github.com/Mehran-TAM/TTH
This work proposes FADE (FFN Attenuation for DEcoding), a training-free method that attenuates FFN outputs to reduce language-prior dominance and shows that FADE effectively mitigates hallucinations while preserving inference efficiency.
Yichen Guo, Kai Tang, Fenglai Lin et al.· arXiv.org· 2 citations
Recent advances in large vision-language models (LVLMs) have enabled powerful multimodal reasoning by integrating visual encoders with large language models (LLMs). However, their reliability is frequently undermined by hallucinations, where generated text inaccurately describes the visual input. Although fine-tuning can mitigate this problem, it is computationally expensive and requires large, curated datasets, making training-free alternatives attractive. Among these, model editing is more promising than decoding-based approaches: decoding methods adapt outputs per input but introduce computational overhead and instability, whereas model editing modifies internal representations offline, providing a more efficient and stable solution. However, existing model-editing techniques typically rely on a single global subspace to correct hallucinations, treating all test samples identically and failing to capture diverse hallucination modes across inputs. To address this limitation, we propose a training-free hallucination mitigation framework for dynamic, per-instance suppression at test time. Our method first constructs a set of Disentangled Hallucination Subspaces, each isolating a distinct hallucination mode. During inference, the model adaptively calculates weights reflecting each input's relationship to these subspaces, guiding a dynamically combined projection that selectively suppresses the most probable hallucination directions while preserving image-grounded semantics. Extensive experiments across multiple vision-language benchmarks and LVLM families demonstrate consistent improvements, highlighting the robustness, generalizability, and efficiency of our approach.
Ali Cheraghian, Hamidreza Dastmalchi, Hamed Barzamini et al.· 0 citations
Despite their widespread use, Large Language Models (LLMs) remain limited by a fundamental problem: the generation of plausible but false content, known as hallucinations. Most existing detection methods operate at the answer or sentence level, yet per-token detection is essential for localizing hallucinated spans and enabling fine-grained interventions. In this paper, we explore the use of the Mixture-of-Experts (MoE) paradigm to address this gap. In MoE architectures, a single forward pass activates a sparse subset of experts (i.e., distinct feedforward networks per layer) via a routing mechanism, producing internal signals (e.g., router entropy, expert disagreement, and expert usage patterns) that are unavailable in dense architectures and have not been previously exploited for hallucination detection. To this end, we introduce InnerExpert, the first method to leverage these MoE-specific signals for per-token hallucination detection. InnerExpert combines routing-level and standard transformer signals into compact per-token feature vectors, classified by a lightweight detector trained on labels produced by an LLM-as-a-judge pipeline, which enables continuous model updates without manual annotation. Our results show that InnerExpert outperforms existing methods across five datasets and two MoE architectures, achieving up to 0.91 answer-level and 0.76 token-level AUROC, while requiring only a single forward pass.
João Fonseca, Rodrigo Rodrigues, Paolo Romano· 0 citations
: When facing real-world scenes that are densely populated with objects or contain complex occlusions, Vision-Language Models are often constrained by the language prior in autoregressive decoding, producing severe hallucination phenomena. To address this pain point that limits the reliable deployment of large models, this paper proposes Dual-branch Inference for Visual-prior Elimination, a training-free hallucination mitigation mechanism for complex scenes. By constructing a dual-branch inference structure at the inference stage and introducing a dynamic visual-confidence penalty, this mechanism effectively quantifies and suppresses the overconfidence in the content generation process, forcing the model’s output to be deeply aligned with the underlying visual features. Results on the object hallucination evaluation benchmark POPE show that, without consuming computing power for model fine-tuning, the proposed method reduces the model’s hallucination rate when objects are dense or complex occlusions exist, and brings a slight improvement in the question-answering accuracy of the model on the MSCOCO and VG datasets.
Shuguo Jiang· Journal of Electronics and I...· 0 citations
Video-Adaptive Debiasing via Evidence Reweighting (VADER), a training-free framework with two complementary modules that combines Contrastive decoding then down-weights predictions that remain confident after selective evidence erasure.
Dong Xing, Jiaxin Chen, Hang Yang et al.· 0 citations
TRE is a parameter-free and single-run metric that estimates hallucination risk directly from the entropy signals of a single generation, without requiring any detector training or repeated sampling, and enjoys strong generalizability, efficiency, and robustness.
Pengcheng Weng, Y. Qian, Yue Tan et al.· 0 citations