Skip to content

Reinforcing What the Student Has Learned: Teacher-Guided On-Policy Distillation

Nov 2026 · 0 citations · 36 references
Computer Science

TL;DR

Student-Delta On-Policy Distillation (SD-OPD), which uses centered student--base differences as update coefficients, keeps each coefficient only when its sign agrees with centered teacher--student feedback, and recomputes the student signal every training round.

Abstract

On-policy distillation (OPD) improves language-model reasoning through dense teacher feedback on student-generated responses. We first observe an amplitude disparity in this feedback: on math reasoning responses, style-related positions exhibit 3.5 times the mean absolute teacher--student log-probability difference of mathematical positions. Scaling OPD updates by these discrepancies thus directs much of the update strength toward matching the teacher's expression. We instead take the student's own pretrained base as a reference: the student--base difference records the token preferences acquired during post-training and provides supervision that does not depend on the student's distance from the teacher. Building on this reference, we propose Student-Delta On-Policy Distillation (SD-OPD), which uses centered student--base differences as update coefficients, keeps each coefficient only when its sign agrees with centered teacher--student feedback, and recomputes the student signal every training round. Across 5 teacher--student configurations with Qwen3 students of 1.7B to 8B parameters, SD-OPD improves six-task mathematics average pass@1 (avg@16) over OPD by 7.76 to 10.37 percentage points. Code is available at https://github.com/aim-uofa/sd-opd.

View source

Similar papers

Calibrating Teacher-Student Discrepancy for On-Policy Distillation

Calibrated On-Policy Distillation is introduced, which estimates the teacher's self-deviation region through positive and negative privileged interventions and calibrates the original teacher--student discrepancy by retaining only the component that lies beyond this region.

Qiang-Qiang He, Jin Li, Ming-Cai Chen · 2 citations · ⚡1
Preprint Nov 2026

Meet Students Where They Are: Student-Prefix Teacher Adaptation for On-Policy Distillation

On-policy distillation (OPD) mitigates the student's exposure bias by optimizing on student-generated trajectories. However, student-generated prefixes may contain reasoning errors or follow patterns unfamiliar to the teacher, undermining the reliability of its token-level supervision. This degradation becomes more pro...

Xue Zhang, Songming Zhang, Yun-Long Liang et al. · 0 citations

Recursive Self-Improvement via On-Policy Distillation for Reasoning

DCE+SRCL outperforms OPSD across multiple model scales and four competition-level mathematics benchmarks, and its comprehensive evaluations show that DCE+SRCL outperforms OPSD across multiple model scales and four competition-level mathematics benchmarks.

Shang-Jian Yin, Ze-Hao Zhao, Kavosh Asadi et al. · 6 citations
#machine learning Preprint Oct 2026

How Should Teachers Be Prepared? RL on Student-Induced States for On-Policy Distillation

On-policy distillation (OPD) improves the reasoning capabilities of small language models through token-level teacher supervision on student-generated trajectories. Yet can teachers that excel at solving problems independently also guide student reasoning effectively? Prior work shows that when student prefixes follow...

Xiao-Yu Ma, Hao-Yue Liu, Zhi-Chao Wang et al. · 0 citations

Beyond Teacher Assignment: Domain-Normalized Multi-Teacher On-Policy Distillation

Domain-Normalized MOPD is proposed, which keeps the routing and rescales each domain's feedback by its measured spread, and improves the average score over MOPD at every size, across three random seeds and under two answer-length limits, and recovers most of the lost mathematics gain.

Xin Li, Hao-Zhe Jiang, Xin-Ming Gao et al. · 2 citations

Learning from a Thoughtful Teacher: Adaptive On-Policy Self-Distillation for Mathematical Reasoning

Adaptive On-Policy Self-Distillation is proposed, which adapts what information the teacher receives and how strongly its feedback influences learning, and encodes each solution as a reasoning DAG, orders problems by the student's evolving capability, and reveals only the affordable subgraph and its next frontier as PI...

Jia-Cheng Du, Wei-Wei Xie, Tian-Yi Du et al. · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.