Skip to content
Preprint

Look Where It Counts: A Free, Label-Free Visual Evidence Signal for Fine-Grained Vision-Language Reasoning

Sep 2026 · 0 citations · 13 references
Computer Science Engineering

TL;DR

The contrastive evidence gap is formalized, the per-token log-likelihood ratio that a model assigns to its own output when conditioned on a question-relevant region versus an irrelevant one, and it is shown that the gap is complementary to the model's own confidence.

Abstract

Multimodal large language models (MLLMs) fail at fine-grained visual questions less because they cannot reason than because they never see the evidence: high-resolution images are downsampled before encoding, so the model answers from linguistic priors. The standard remedies are expensive: annotated answers (SFT), hand-engineered verifiers (RLVR), or a large external teacher (on-policy distillation). We ask whether the visual evidence itself can supply the signal for free. We formalize the contrastive evidence gap, the per-token log-likelihood ratio that a model assigns to its own output when conditioned on a question-relevant region versus an irrelevant one, and study it across Qwen2.5-VL-7B, Qwen3-VL-8B, and Qwen3-VL-30B-A3B on V*Bench. Our main positive result is training-free: selecting the candidate crop under which the model's answer distribution is most peaked, using a single-view, label-free criterion, discovers the answer-bearing region with no bounding boxes, training, or labels. It localizes the target 4.4 to 5.1 times better than chance and raises fine-grained accuracy from 70 percent to 85 percent at inference. We further show that the gap is complementary to the model's own confidence. Combining them predicts correctness better than either alone, with AUC up to 0.99, and flags confidently wrong answers, with AUC ranging from 0.97 to 1.00 within the high-confidence subset. All effects concentrate on perception-bottleneck questions and vanish on a global-context control. Finally, we report an honest negative result: converting the same signal into a training method, gated self-distillation (SEG-Distill), does not outperform the base model at pilot scale across three gate designs, while more aggressive gating degrades accuracy. The signal is real, but converting it into training gains remains an open problem.

View source

Similar papers

Sep 2026

Beyond Saying Less: Fine-Grained Alignment for Informative and Faithful Vision-Language Models

This work proposes a fine-grained alignment framework that couples dense reward signals at the data level with precise credit assignment at the algorithmic level to prevent response-level shared advantages from allowing local hallucinations to compromise all other valid outputs within the same response.

Xing-Ming Long, Jie Zhang, Yue-Cong Min et al. · 0 citations
#computer vision Review Aug 2026

Learning to Look Again: Loss-Gap Supervision for Free-form Crop Routing in Vision-Language Models

GapSight is proposed, a framework for learning visual re-reading: a VLM first takes a global glance, then selectively returns to a free-form region when the question calls for local evidence, and Mechanism analyses show that the router rescues concrete wrong answers, adapts its action rate by task, and forms a favorabl...

Jin-Chang Zhu, Rong Fu, Yi-Cheng Ding et al. · 1 citation
Preprint Sep 2026

CrowdCue: Specialist-Cue Conditioning for Vision-Language Crowd Counting

Generative vision-language models (VLMs) offer a counting paradigm in which one model produces both a count and a natural-language account of the scene, yet their raw counting accuracy sits in the range of sub-million-parameter specialist regressors. The open question is whether auxiliary guidance from a pretrained spe...

M. Farazi, B. Ciftler, Abdulhalim Dandoush et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Cross-Modal Attention Acts as a Frequency Filter: Why Verbose Prompts Improve Robustness in Vision-Language Models

This work finds that the wording of the question affects VLMs in two opposite ways: verbose paraphrasing reduces drift variance by 70--81% on the 8B models and the practical recipe---pad the prompt---further yields measurable gains in accuracy, even under image corruption.

Farooq Ahmad Wani, Maria Sofia Bucarelli, Mujtaba Hussain Mirza et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.