Mitigating Hallucinations in Large Vision-Language Models With Fine-Grained Cross-Modal Attention
Abstract
Large Vision–Language Models (LVLMs) achieve strong performance on many multimodal tasks but still suffer from object hallucinations, where they confidently describe non-existent or visually unsupported objects. Recent studies suggest that such errors are closely linked to imbalanced cross-modal attention: during decoding, LVLMs often over-rely on textual priors and fail to attend to query-relevant visual evidence. By revisiting internal attention patterns, we empirically observe that, except for the very early layers, attention heads with higher image-attention scores tend to concentrate on query-relevant regions. Motivated by this finding, we propose Fine-grained Cross-modal Attention (FCA), a training-free and tool-free decoding framework that mitigates hallucinations by directly manipulating the LVLM's own cross-modal attention. Concretely, FCA locates the layer and heads that place the strongest focus on image tokens, aggregates their cross-modal attention into a joint map, and selects a compact set of query-relevant image tokens via a cumulative-attention criterion. It then injects token-level masks at selected layers and performs attention-guided dual-branch decoding under an adaptive plausibility constraint, selectively amplifying grounded visual evidence while suppressing hallucination-prone predictions. Extensive experiments on hallucination benchmarks (i.e., POPE, CHAIR, and MME) and standard VQA datasets (i.e., VizWiz, MM-Vet, GQA, and MMBench) demonstrate that FCA substantially reduces hallucinations while maintaining or even improving overall VQA performance.