Chao-Hong Tan

LoOPSD: On-Policy Self-Distillation with Looped Teachers

On-policy self-distillation improves a model using guidance from the model itself, but it still requires a teacher that provides more useful predictions than the student. Existing approaches obtain such guidance by giving the teacher additional context or instructions. We ask whether it can instead come from changing h...