Skip to content

Not Every Rollout Needs Distillation: Correctness-Filtered On-Policy Self-Distillation

Oct 2026 · 0 citations · 25 references
Computer Science

TL;DR

Correctness-filtered OPSD (C-OPSD), which generates one response per problem and distills only a subset of incorrect responses, is introduced, which connects outcome-conditioned supervision redundancy to more compute-efficient distillation.

Abstract

On-policy self-distillation (OPSD) supervises student-generated trajectories with a reference-conditioned version of the same base model. We ask whether every trajectory supplies distinct supervision that warrants its distillation cost. Gradient measurements on Qwen3-1.7B at initialization reveal that correct and incorrect responses have similar individual gradient magnitudes, but incorrect responses have approximately three times the coherent gradient share. Their estimated population-mean directions also largely overlap. A signal--noise model fitted at a small group size tracks alignment at larger group sizes without refitting. Motivated by these measurements, we introduce correctness-filtered OPSD (C-OPSD), which generates one response per problem and distills only a subset of incorrect responses. Removing unselected rows before both distillation forwards avoids their teacher/student forward and student backward work. Across Qwen3-1.7B, 4B, and 8B on AIME24, AIME25, and HMMT25, C-OPSD achieves on-par performance with full OPSD. In a measured micro-batch-size-four implementation, executed rows decrease by approximately 68%, and with retained rows packed into full micro-batches, training takes about 40% less time while rollout generation remains unchanged. Together, the measurements and training results connect outcome-conditioned supervision redundancy to more compute-efficient distillation.

View source

Similar papers

Activation-Conditioned Self-Distillation

Activation-Conditioned Self-Distillation (ACSD), which extracts a steering vector by contrasting activations of self-generated trajectories that reach verified correct answers within a generation budget with those of all remaining trajectories, achieves the highest mean accuracy over four mathematical benchmarks among...

Zhe-Xi Lu, Subhajit Chaudhury, Tejaswini Pedapati et al. · 1 citation
#artificial intelligence Preprint Oct 2026

Does On-Policy Distillation Need to Complete Every Rollout?

On-policy distillation (OPD) transfers teacher knowledge to a student through feedback on student-generated rollouts, but generating complete rollouts and computing per-token teacher feedback are computationally expensive. Existing truncation methods save computation by stopping rollouts early. However, they do not com...

Zheng-Xin Zhang, Mao-Sen Tang, Ning Wang et al. · 0 citations
#natural language process... Preprint Oct 2026

When Do We Need On-Policy Distillation? Distilling on Offline Student Rollouts Is Often Better

On-policy distillation (OPD) has become increasingly popular for transferring teacher capabilities to student models. In this work, we ask a critical research question: Is on-policy sampling always beneficial for distilling arbitrary teacher-student pairs? We show that a simple alternative, Semi-OPD, which distills fro...

Siyan Zhao, Yong-Gan Fu, Jin-Dong Jiang et al. · 2 citations

Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning

On-policy self-distillation (OPSD) trains a student to match a privileged teacher distribution along its own sampled trajectory. Standard OPSD applies this supervision to unverified student rollouts while conditioning the teacher on privileged context, typically a reference solution. We separate these roles in a factor...

Md. Ismail Hossain, Humaira Kousar, Isidora Chara Tourni · 1 citation
Preprint Oct 2026

Adaptive-Policy Distillation: Knowledge Injection under Information Asymmetry

On-policy distillation (OPD), which uses reverse-KL supervision on student rollouts to transfer a context-conditioned model's behavior into student model parameters for closed-book inference, has shown strong results. However, we find that this success does not translate to knowledge injection: across two document-QA b...

Hojun Cho, K. Jeon, Jaegul Choo · 0 citations
Preprint Oct 2026

Asynchronous On-Policy Self-Distillation

This work introduces Asynchronous On-Policy Self-Distillation (AsyncOPSD), a framework that decouples rollout generation from learner updates to improve training throughput and introduces a distillation objective based on staleness-aware policy control and token-level entropy weighting.

Xi-Dong Wu, Qi Qi, Yu-Qiong Ji et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.