Skip to content
Book Open access

The Modality Neglect Problem: Measuring Visual Reliance in Vision-Language Models

Nov 2026 · 0 citations · 14 references

Abstract

How much do Vision-Language Models (VLMs) actually depend on their visual input? We introduce the Visual Reliance Score (VRS) and a five-condition behavioral testing framework to rigorously answer this question. By evaluating each query under standard, blank-image, random-image, contradictory-image, and image-only conditions across three image-essential benchmarks (DocVQA, ChartQA, TextVQA), we produce a behavioral fingerprint revealing when and how models engage with, or neglect, visual information. Across 24,000 inferences spanning two model families (Qwen2.5-VL, InternVL2) and two scales (7B and 72B), we find: (1) visual reliance is high on image-essential benchmarks (VRS 0.85-0.98) but varies significantly across tasks; (2) modality neglect persists even on these benchmarks, with up to 24% of answers unchanged when the image is removed, and most neglected answers being correct, revealing shortcut reliance on language priors; (3) scaling reduces neglect: the 72B model shows 1.0% average neglect vs. 15.6% for the 7B model, indicating that larger models are more visually grounded; and (4) a forced grounding intervention reduces neglect by 47% on ChartQA, though the effect does not transfer to less-structured scene-text tasks. We propose VRS as a mandatory companion metric for VLM evaluation and provide a taxonomy of neglect types (benign, harmful, and catastrophic) with implications for both benchmark design and training methodology.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.