Preprint

Perception Before Supervision: Self-Contained Visual Distillation from Counterfactual Blind Spots

Shravan Venkatraman Omkar Thawakar Ritesh Thawkar Abdelrahman M. Shaker R. Anwer

Aug 2026 · 1 citation · 42 references

Computer Science

TL;DR

This work introduces CVPD (Contrastive Counterfactual Visual Process Distillation), which is the first fully self-contained framework for dense, on-policy, token-level visual self-distillation for MLLMs, and proposes a three-gate Counterfactual Criterion that identifies visual blind spots where zooming into a region changes and sharpens the model's answer distribution.

Abstract

Self-improvement for multimodal large language models (MLLMs) is typically driven by reward-based methods that provide only coarse scalar feedback. Distillation offers a richer alternative through dense token-level supervision, but in the visual domain it usually depends on privileged context constructed using external annotations and tools, or stronger models. We introduce \textbf{CVPD} (Contrastive Counterfactual Visual Process Distillation), which, to the best of our knowledge, is the first fully self-contained framework for dense, on-policy, token-level visual self-distillation for MLLMs. CVPD identifies visual blind spots where zooming into a region changes and sharpens the model's answer distribution, while removing the same region leaves the full-image behavior largely unchanged. Such regions reveal perceptual information that the model can encode but fails to consistently utilize under full-image conditioning. We propose a three-gate Counterfactual Criterion that identifies these regions directly from the model's own responses and converts them into dense contrastive supervision for self-distillation. On Qwen3-VL-8B-Instruct, CVPD outperforms six self-evolving baselines across twelve benchmarks, including methods that rely on external GPT-4o supervision, without a single regression. It achieves gains of $+3.60$ on OCRBench, $+3.38$ on MMStar Fine-Grained Perception, and $+3.08$ on MMStar Logical Reasoning, while maintaining or improving performance on broader multimodal benchmarks.

arXiv Semantic Scholar

View source

Similar papers

Preprint Aug 2026

OPD-V: Visual On-Policy Self-Distillation with Modality Balance

OPD-V is introduced, a visual OPSD paradigm that instantiates privileged information through the Positive Teacher and Negative Teacher that consistently improves reasoning performance while reducing training cost.

Aniri, Jinhe Bi, Peng Liao et al. · 2 citations

Jul 2026

Visual Contrastive Self-Distillation

On-policy self-distillation (OPSD) is promising as it removes the external teacher required by on-policy distillation (OPD), yet it still needs asymmetric information between teacher and student to ensure that the self-teacher provides a stronger learning signal than the student. Existing methods create this asymmetry either through privileged answers or visual evidence. We ask whether both can be removed, yielding a simpler form of OPSD driven purely by input conditioning. For this purpose, we propose Visual Contrastive Self-Distillation, namely VCSD, which converts image-content removal into an on-policy self-distillation signal. At each student-generated response prefix, the EMA teacher produces two next-token distributions under the same prompt and prefix -- one conditioned on the original image and the other on a content-erased control. Their token-wise log-probability difference highlights candidates whose likelihood is specifically increased by the instance-level visual content. We use this contrast to sharpen the teacher's original-image distribution within its plausible support, and distill the resulting full-distribution target into the student. Using ViRL39K dataset, VCSD consistently outperforms matched OPSD across Qwen3-VL and Qwen3.5 models. For example, on Qwen3-VL, it improves the seven-benchmark aggregate from $62.27\% \rightarrow 67.04\%$ at 2B, $71.30\% \rightarrow 73.16\%$ at 4B, and $72.51\% \rightarrow 76.26\%$ at 8B. Furthermore, VCSD requires no external teacher, privileged answers, visual evidence signals, reasoning traces, or additional inference-time cost.

Yijun Liang, Yunjie Tian, Yijiang Li et al. · arXiv.org · 4 citations

Jul 2026

Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

For the coarse attributes the authors study, MLLMs encode the visual evidence but cannot reliably control their reliance on it, indicating that for the coarse attributes they study, MLLMs cannot reliably control their reliance on it.

Jiaang Li, Chengzu Li, Zhaochong An et al. · arXiv.org · 0 citations

Preprint Aug 2026

Self-Supervised Visual On-Policy Distillation

Self-Supervised Visual On-Policy Distillation (S$^2$VOPD), a simple yet effective method that constructs on-policy learning signals from asymmetric augmented views, systematically explores a broad design space of visual augmentations and uncover that asymmetry matters.

Yijiang Li, Yijun Liang, Yunjie Tian et al. · 0 citations

Preprint Aug 2026

Credit the Right Box: Marginal Contribution Assignment for Structured Visual Perception

This work proposes MCR-GRPO, a marginal contribution assignment framework that derives box-level credit directly from each sampled response, preserving GRPO's response-level comparison while enabling box-aware optimization of structured multi-object grounding.

Xinheng Han, Jianfeng Wang, Yu Chen et al. · 0 citations

Jul 2026

Self-Boosting Vision-Language Models with Noisy Student On-Policy Self-Distillation

This work proposes NOPD (Noisy Student On-Policy Self-Distillation), a simple yet effective self-distillation approach that improves VLMs without any external models or ground-truth answers, and demonstrates that NOPD is a general approach to enhance VLMs, achieving improvements across three models on 12 benchmarks.

Shuai Wang, Daoan Zhang, Zhe Tang et al. · arXiv.org · 0 citations