Jul 2026
Weak-to-Strong On-Policy Distillation
W2S-OPD is introduced, a simple yet effective OPD framework that improves the strong student by distilling from multiple weak models, and outperforms OPD, enables the student to surpass the domain teacher, and keeps improving the student even when every supervision source is weaker.
Fangxu Yu, Zinan Lin, Xiaodong Liu et al.
· arXiv.org · 7 citations
· ⚡1