On-Policy Transition Distillation
A diagnostic study finds that stronger teacher--student agreement in intermediate transitions is associated with higher answer correctness within questions, with correct responses exhibiting lower mean transition KL across all measured intermediate layer pairs.
Run-Yang You, Yong-Qi Li, Tie-Zheng Yu et al.
· 0 citations