Adaptive Pullback: Reweighting Supervision at Penalty Jumps in On-Policy Distillation
On-policy distillation (OPD) supervises student-generated responses using sampled-token student-teacher log-ratios. A positive log-ratio penalizes a token that the student assigns higher probability to than the teacher, but its current value does not distinguish sustained high penalties from abrupt increases along a re...
Guang-Yu Liu, Si-Yuan Gan, Shi-Long Zhou et al.
· 0 citations