Preprint

Rethinking Reverse KL as Adaptive Entropy Distillation

Shizheng Li Zhiyu Shen Yu-Yin Lu Yunhe Pang Jielin Song Yanghui Rao Fu-Lee Wang

Aug 2026 · 0 citations · 32 references

Computer Science Mathematics

TL;DR

This work revisits on-policy Reverse Kullback-Leibler distillation and decomposes its objective into a teacher-fitting term and a student-entropy term, without introducing an explicit FKL branch, and proposes Adaptive Entropy Distillation (AED), which uses the teacher's entropy to dynamically calibrate token-level imitation strength.

Abstract

Knowledge distillation (KD) is widely used to transfer the capabilities of large language models (LLMs) to smaller students, but existing objectives often struggle to balance faithful imitation and robust generation. In particular, existing methods mainly combine FKL and RKL, overlooking that RKL itself provides a mechanism for adjusting the student's imitation strength. Motivated by this, we revisit on-policy Reverse Kullback-Leibler (RKL) distillation and decompose its objective into a teacher-fitting term and a student-entropy term, without introducing an explicit FKL branch. We show theoretically that the token-level optimal student distribution corresponds to a tempered variant of the teacher distribution, where the adaptive weight controls the trade-off between mode-seeking and uncertainty preservation. Guided by this insight, we propose \textbf{Adaptive Entropy Distillation (AED)}, which uses the teacher's entropy to dynamically calibrate token-level imitation strength. Experiments on instruction-following and mathematical reasoning benchmarks demonstrate that AED achieves superior overall performance and generally improves teacher--student distributional and entropy alignment.

arXiv Semantic Scholar

View source

Similar papers

#natural language process... Preprint Jul 2026

Geometric Self-Distillation for Reasoning Generalization

GeoSD, a geometric self-distillation objective that treats this drift as movement in the student's predictive behavior and counters it in two complementary ways, preserves the in-distribution gains of self-distillation while improving average OOD accuracy.

Josip Jukic, Ivan Titov · 0 citations

#machine learning Jul 2026

Distilled Reinforcement Learning for LLM Post-training

Extensive experiments show that Distilled RL substantially outperforms standard RL and OPD in terms of both pass@1 and pass@k, and can effectively transfer previously unavailable knowledge from a teacher model to a student model.

Chen Wang, Zhaochun Li, Jionghao Bai et al. · arXiv.org · 2 citations

Jul 2026

Contrastive Reinforced Policy Optimization via Privileged Self-Distillation

Contrastive Reinforced Policy Optimization (CRPO) is introduced, which reformulates agentic OPSD from a contrastive learning perspective, and conducts group-wise contrast to preserve reliable, fine-grained optimization signals.

Xingjian Wu, Junlin Liu, Xing-Chen Liu et al. · arXiv.org · 2 citations

Jul 2026

Reward-Gated On-Policy Distillation

RG-OPD bridges sparse verifier rewards and dense teacher logits, preserving token-level supervision while filtering misleading teacher signals, and produces stronger distilled students, outperforming both vanilla reverse-KL distillation and the recent TSD-KD baseline.

Mohammad Sadegh Akhondzadeh, Vijay Lingam, Atula Tejaswi et al. · arXiv.org · 4 citations · ⚡1

Preprint Jul 2026

dOPSD: On-Policy Self-Distillation for Diffusion Language Models

dOPSD derives the teacher's privilege directly from the student's own denoising trajectory, evaluating masked positions using later, more-decoded steps of that same trajectory rather than an external label, so the teacher's advantage emerges from the model's own decoding process.

P. Dat, Qi Li, Xinchao Wang · 0 citations

Book Open access Aug 2026

SDE : Scale-Difference Evolution Knowledge Distillation

Logit-based knowledge distillation (KD) is pivotal for efficient model compression and cross-architecture learning. However, conventional methods typically rely on static, single-scale logit alignment, thereby overlooking the semantic evolution trajectory embedded in cross-scale prediction transitions. To bridge this g...

Hejie Lu · Proceedings of the 32nd ACM... · 0 citations