Skip to content
Open access

Mitigating Hallucinations in Large Vision-Language Models With Fine-Grained Cross-Modal Attention

2026 · IEEE transactions on multimedia · Vol 28, pp. 17474-17487 · 0 citations · 69 references

Abstract

Large Vision–Language Models (LVLMs) achieve strong performance on many multimodal tasks but still suffer from object hallucinations, where they confidently describe non-existent or visually unsupported objects. Recent studies suggest that such errors are closely linked to imbalanced cross-modal attention: during decoding, LVLMs often over-rely on textual priors and fail to attend to query-relevant visual evidence. By revisiting internal attention patterns, we empirically observe that, except for the very early layers, attention heads with higher image-attention scores tend to concentrate on query-relevant regions. Motivated by this finding, we propose Fine-grained Cross-modal Attention (FCA), a training-free and tool-free decoding framework that mitigates hallucinations by directly manipulating the LVLM's own cross-modal attention. Concretely, FCA locates the layer and heads that place the strongest focus on image tokens, aggregates their cross-modal attention into a joint map, and selects a compact set of query-relevant image tokens via a cumulative-attention criterion. It then injects token-level masks at selected layers and performs attention-guided dual-branch decoding under an adaptive plausibility constraint, selectively amplifying grounded visual evidence while suppressing hallucination-prone predictions. Extensive experiments on hallucination benchmarks (i.e., POPE, CHAIR, and MME) and standard VQA datasets (i.e., VizWiz, MM-Vet, GQA, and MMBench) demonstrate that FCA substantially reduces hallucinations while maintaining or even improving overall VQA performance.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.