Beyond Data Schedule: Fast LLM Pretraining on Data Mixtures by Loss-Reweighting
Large language models often acquire specialized capabilities through successive pretraining stages, but these gains can come at the cost of forgetting earlier knowledge. In this work, we investigate whether such trade-offs can be improved by changing how strongly each domain contributes to learning while keeping its da...
Jia-Zheng Li, Hua-Qing Zhang, Jing Xu et al.
· 0 citations