Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation
TrustMOPD replaces example-level teacher selection with label-free, token-level supervision allocation and closes the overall-score gap between the initial student and oracle-routed teachers when trained on single- and multi-teacher on-policy distillation.
Jie Sun, Mao Zheng, Ming-Yang Song et al.
· 6 citations