Estimating and Orthogonalizing Unknown Pre-training Gradients for Continual Fine-tuning of Large Language Models
Continual fine-tuning is essential for large language models (LLMs) to dynamically adapt to real-world environments, yet it inevitably suffers from catastrophic forgetting, particularly the performance degradation of previous tasks and LLMs'general-purpose knowledge. Although existing methods, such as orthogonal gradie...
Bing Wang, C. Li, Xin-Qiang Cai et al.
· 0 citations