OPOD: On-Policy Omni Distillation

Tong Zhao Yuyang Hu Reed Li Yu Lu Haibo Shi Yutao Zhu Zhicheng Dou

Jul 2026 · arXiv.org · Vol abs/2607.20918 · 0 citations · 41 references

Computer Science

TL;DR

On-Policy Omni Distillation (OPOD), which consolidates text, image, and audio teachers into one omni model, and surpasses the base model and pooled RL training on all twelve benchmarks, and ranks first or second on eleven even when the teachers are included.

Abstract

Omni-modal models provide a unified interface for text, images, and audio. However, improving these abilities together remains difficult, as post-training on pooled multimodal data often fails to preserve the strengths of modality teachers. On-policy distillation (OPD) has recently become popular in model post-training. It samples responses from the current student and compares the teacher's and student's next-token distributions along those responses, yielding dense supervision while reducing the mismatch between training and inference. Despite these advantages, standard OPD does not readily extend to several modality teachers. Their guidance may favor conflicting changes to the shared model, while matching each teacher's next-token distribution can prevent the student from moving beyond that teacher. To address these challenges, we propose On-Policy Omni Distillation (OPOD), which consolidates text, image, and audio teachers into one omni model. OPOD routes each response to the corresponding teacher, controls the teachers independently, and applies guidance only when the teacher assigns a higher probability to the generated token. The selected teacher also evaluates answer confidence and whether the reasoning increases support for the answer. Extensive experiments on twelve benchmarks show that OPOD achieves the best average at three model scales, reaching 70.8, 51.7, and 46.2 and outperforming the strongest comparator by 2.1, 1.8, and 1.7 points. At 30B, it surpasses the base model and pooled RL training on all twelve benchmarks, and ranks first or second on eleven even when the teachers are included. Only the student is retained for deployment.

DOI arXiv Semantic Scholar

View source

Similar papers

Jul 2026

RP-OPSD: Resolution-Privileged On-Policy Self-Distillation for Multimodal Large Language Models

It is demonstrated that resolution differences can serve as a simple and scalable source of privileged information, providing an effective and efficient approach to on-policy self-distillation for multimodal large language models.

Qihui Zhu, Yucheng Wang, Zijian Wen et al. · arXiv.org · 0 citations

Preprint Aug 2026

OPD-V: Visual On-Policy Self-Distillation with Modality Balance

OPD-V is introduced, a visual OPSD paradigm that instantiates privileged information through the Positive Teacher and Negative Teacher that consistently improves reasoning performance while reducing training cost.

Aniri, Jinhe Bi, Peng Liao et al. · 2 citations

Preprint Aug 2026

Poly-OPD: Heterogeneous Multi-Teacher On-Policy Distillation for Capability-Selectable Flow Models

This paper presents Poly-OPD, a framework that can consolidate complementary strengths of heterogeneous teachers into a single compact flow-matching student, and uses a gradient compatibility diagnostic to organize its adapters.

Siming Fu, Haojun Xu, R.Z. He et al. · 2 citations

Preprint Aug 2026

Self-OPD: On-Policy Distillation for Flow Matching Models without Teacher

Self-OPD is introduced, a teacher-free OPD framework for flow matching models that turns the student's own self-exploration into step-wise supervision and outperforms prior RL and OPD methods without task-specific teachers.

Shiyi Zhang, Mu-Shui Liu, Yunze Tong et al. · 0 citations

Jul 2026

Visual Contrastive Self-Distillation

On-policy self-distillation (OPSD) is promising as it removes the external teacher required by on-policy distillation (OPD), yet it still needs asymmetric information between teacher and student to ensure that the self-teacher provides a stronger learning signal than the student. Existing methods create this asymmetry either through privileged answers or visual evidence. We ask whether both can be removed, yielding a simpler form of OPSD driven purely by input conditioning. For this purpose, we propose Visual Contrastive Self-Distillation, namely VCSD, which converts image-content removal into an on-policy self-distillation signal. At each student-generated response prefix, the EMA teacher produces two next-token distributions under the same prompt and prefix -- one conditioned on the original image and the other on a content-erased control. Their token-wise log-probability difference highlights candidates whose likelihood is specifically increased by the instance-level visual content. We use this contrast to sharpen the teacher's original-image distribution within its plausible support, and distill the resulting full-distribution target into the student. Using ViRL39K dataset, VCSD consistently outperforms matched OPSD across Qwen3-VL and Qwen3.5 models. For example, on Qwen3-VL, it improves the seven-benchmark aggregate from $62.27\% \rightarrow 67.04\%$ at 2B, $71.30\% \rightarrow 73.16\%$ at 4B, and $72.51\% \rightarrow 76.26\%$ at 8B. Furthermore, VCSD requires no external teacher, privileged answers, visual evidence signals, reasoning traces, or additional inference-time cost.

Yijun Liang, Yunjie Tian, Yijiang Li et al. · arXiv.org · 4 citations

#small language model Preprint Aug 2026

Video-OPSD: Exploiting Privileged Visual Evidence for On-Policy Self-Distillation in Video Large Language Models

Experiments across video understanding and reasoning benchmarks show that the Evidence-Grounded Self-Teacher framework consistently improves upon Standard OPSD across multiple backbones and achieves performance comparable to GRPO while requiring substantially less training time, establishing an effective and efficient post-training approach for Video-LLMs.

Zi-Yue Wang, Shiqi Huang, Wei-Wen Xu et al. · 0 citations