Slow-Fast Multi-Teacher On-Policy Distillation for Capability Preservation
Slow-Fast Multi-Teacher On-Policy Distillation (SF-MOPD), which couples a fast model, the current student updated directly by each teacher, with a slow model, an exponential moving average of the student, which effectively mitigates capability interference, enhances specialized multimodal capabilities, and reduces the...
Xiao-Fei Yin, Tong Chu, Ji-Yuan Fu et al.
· 0 citations