RP-OPSD: Resolution-Privileged On-Policy Self-Distillation for Multimodal Large Language Models

Qihui Zhu Yucheng Wang Zijian Wen Tao Zhang Meng Zhang Yang Liu Shuang-Wu Chen Si-Ying Wu Jian Yang Xiaofeng Jiang

Jul 2026 · arXiv.org · Vol abs/2607.24447 · 0 citations · 22 references

Computer Science

TL;DR

It is demonstrated that resolution differences can serve as a simple and scalable source of privileged information, providing an effective and efficient approach to on-policy self-distillation for multimodal large language models.

Abstract

On-Policy Self-Distillation (OPSD) uses privileged information available only to the teacher to provide dense token-level supervision on trajectories generated by the student. However, existing methods often rely on verified solution traces, explanations generated by external models, or manually localized visual evidence, which limits their scalable application to multimodal large language models. To address this issue, we exploit the information gap between high- and low-resolution views of the same image and propose RP-OPSD (Resolution-Privileged On-Policy Self-Distillation for Multimodal Large Language Models). During training, the student policy generates on-policy trajectories from images at one-quarter of the original resolution, while the teacher policy provides supervision using the original-resolution images. By minimizing the divergence between their output distributions along the student trajectories, the student learns the predictive behavior of the teacher under high-resolution inputs, thereby strengthening its low-resolution capability and transferring the learned improvement to original-resolution inference. RP-OPSD requires neither additional human annotations nor external models to generate solution traces but only image--question pairs. Experiments on Qwen3.5-9B show that RP-OPSD achieves a 5.45\% relative improvement in average performance at the original resolution and a $1.78\times$ training speedup over OPSD. These results demonstrate that resolution differences can serve as a simple and scalable source of privileged information, providing an effective and efficient approach to on-policy self-distillation for multimodal large language models.

DOI arXiv Semantic Scholar

View source

Similar papers

Preprint Aug 2026

On-Policy Self-Distillation without Any Supervision

U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold, and conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong.

Yijiang Li, Bingyang Wang, Yijun Liang et al. · 2 citations

Preprint Aug 2026

Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher

Self-OPD is introduced, a teacher-free OPD framework for flow matching models that turns the student's own self-exploration into step-wise supervision and outperforms prior RL and OPD methods without task-specific teachers.

Shiyi Zhang, Mu-Shui Liu, Yunze Tong et al. · 0 citations

#small language model Preprint Aug 2026

Video-OPSD: Exploiting Privileged Visual Evidence for On-Policy Self-Distillation in Video Large Language Models

Experiments across video understanding and reasoning benchmarks show that the Evidence-Grounded Self-Teacher framework consistently improves upon Standard OPSD across multiple backbones and achieves performance comparable to GRPO while requiring substantially less training time, establishing an effective and efficient post-training approach for Video-LLMs.

Zi-Yue Wang, Shiqi Huang, Wei-Wen Xu et al. · 0 citations

Preprint Aug 2026

SR-OPSD: Self-Referenced On-Policy Self-Distillation

Extensive experiments across scientific evaluation, mathematical reasoning, and coding generation tasks with multiple large language models show that SR-OPSD achieves the state-of-the-art or competitive performance across various settings.

Zhuo Sun, Entong Li, Yan-Long Zhao et al. · 0 citations

Preprint Aug 2026

OPD-V: Visual On-Policy Self-Distillation with Modality Balance

OPD-V is introduced, a visual OPSD paradigm that instantiates privileged information through the Positive Teacher and Negative Teacher that consistently improves reasoning performance while reducing training cost.

Aniri, Jinhe Bi, Peng Liao et al. · 2 citations

#artificial intelligence Preprint Sep 2026

Rethinking On-Policy Distillation of Large Language Models II: One Training Example

On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. We examine this role at the data-minimal limit by training on a single query. One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families. We explain this result through the states visited during training and the rate at which the student aligns with the teacher. We measure \emph{state coverage}, the fraction of the states full-data OPD visits that a query set's rollouts reach. A single query already reaches $71.5\%$, most of it within the first 100 steps. Adding semantically distinct queries raises coverage and validation accuracy together, until 16 queries reach $98.9\%$ and match full-data training. Yet alignment slows at a similar pace whether OPD trains on one query or the whole dataset, and even a fixed set of states takes hundreds of steps to absorb. OPD is therefore data-overfed but algorithm-starved. Its rollouts quickly expose broad supervision, while the student absorbs that supervision increasingly slowly. The state-coverage result extends to multi-teacher OPD, where 16 semantically diverse queries per domain match full-data MOPD. As a further stress test, content-light templates and off-domain WildChat queries also approach the real-query baseline. Task content and induced state coverage can therefore come apart. We hope these findings direct future work toward the step efficiency of OPD, and prompt a re-examination of the data and the mechanisms behind its recent successes in frontier post-training.

Zi-Xuan Fu, Bing-Xiang He, Yu-Xin Zuo et al. · 1 citation