iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs
On-policy self-distillation fine-tuning (SDFT) learns new skills from demonstrations while reducing forgetting, but it always distils toward the full demonstration-conditioned teacher. This fixes teacher influence at the full-teacher endpoint, providing no control over how much demonstration information should be trans...
Ahmed Khaled Khamis, Xiao-Tong Ji, Hassan Jaber et al.
· 0 citations