Recovering General Capabilities via Uncertainty-Calibrated Multi-Teacher On-Policy Distillation
Specializing large language models to vertical domains improves domain-specific behavior but often degrades general capabilities. We study this trade-off in Multi-Teacher On-Policy Distillation (MOPD), where a specialized model learns from domain and general teachers on its own sampled trajectories. Standard MOPD faces...
Zi-Yuan Liu, Jiao Ou, Jian Liang et al.
· 0 citations