DIVE: A Training-free Hallucination Mitigation Mechanism for Complex Scenes
TL;DR
Results on the object hallucination evaluation benchmark POPE show that the proposed Dual-branch Inference for Visual-prior Elimination reduces the model’s hallucination rate when objects are dense or complex occlusions exist, and brings a slight improvement in the question-answering accuracy of the model on the MSCOCO and VG datasets.
Abstract
: When facing real-world scenes that are densely populated with objects or contain complex occlusions, Vision-Language Models are often constrained by the language prior in autoregressive decoding, producing severe hallucination phenomena. To address this pain point that limits the reliable deployment of large models, this paper proposes Dual-branch Inference for Visual-prior Elimination, a training-free hallucination mitigation mechanism for complex scenes. By constructing a dual-branch inference structure at the inference stage and introducing a dynamic visual-confidence penalty, this mechanism effectively quantifies and suppresses the overconfidence in the content generation process, forcing the model’s output to be deeply aligned with the underlying visual features. Results on the object hallucination evaluation benchmark POPE show that, without consuming computing power for model fine-tuning, the proposed method reduces the model’s hallucination rate when objects are dense or complex occlusions exist, and brings a slight improvement in the question-answering accuracy of the model on the MSCOCO and VG datasets.