The Modality Neglect Problem: Measuring Visual Reliance in Vision-Language Models
Abstract
How much do Vision-Language Models (VLMs) actually depend on their visual input? We introduce the Visual Reliance Score (VRS) and a five-condition behavioral testing framework to rigorously answer this question. By evaluating each query under standard, blank-image, random-image, contradictory-image, and image-only conditions across three image-essential benchmarks (DocVQA, ChartQA, TextVQA), we produce a behavioral fingerprint revealing when and how models engage with, or neglect, visual information. Across 24,000 inferences spanning two model families (Qwen2.5-VL, InternVL2) and two scales (7B and 72B), we find: (1) visual reliance is high on image-essential benchmarks (VRS 0.85-0.98) but varies significantly across tasks; (2) modality neglect persists even on these benchmarks, with up to 24% of answers unchanged when the image is removed, and most neglected answers being correct, revealing shortcut reliance on language priors; (3) scaling reduces neglect: the 72B model shows 1.0% average neglect vs. 15.6% for the 7B model, indicating that larger models are more visually grounded; and (4) a forced grounding intervention reduces neglect by 47% on ChartQA, though the effect does not transfer to less-structured scene-text tasks. We propose VRS as a mandatory companion metric for VLM evaluation and provide a taxonomy of neglect types (benign, harmful, and catastrophic) with implications for both benchmark design and training methodology.