Entropy Interventions Do Not Transfer: Why RLVR Entropy Control Misfires in On-Policy Distillation

Abstract

Reinforcement learning with verifiable rewards (RLVR) controls policy entropy by reweighting tokens against an external scalar advantage. On-policy distillation (OPD) faces the same surface problem, a student that can sharpen too fast, but it replaces that advantage with a per-token teacher target. That shared symptom makes an RLVR entropy fix look reusable. We show the reuse fails for a simple reason. Re-deriving the token-level entropy change of the OPD objective shows its governing factor is the covariance between the teacher-student probability gap and the log policy, not an advantage. The two decompositions disagree on which tokens drive entropy, so transplanting an RLVR entropy reweighting such as STEER into OPD damps exactly the teacher-corrective tokens the student must learn. A preliminary API analysis makes this failure mode visible before training, with corrective tokens over-represented in the high-entropy tail a STEER-style reweight suppresses. A controlled four-seed distillation run then confirms the prediction. Transplanting STEER collapses GSM8K accuracy from 38% to 7%, with corrective-token retention falling in step, while the MATH-500 effect stays within seed noise. We propose a corrected reweighting that reads the teacher-student sign before damping. It restores corrective-token retention and returns to the vanilla OPD band on GSM8K within seed noise.