L. Mackey

Domain-Aware Scaling Laws Uncover Data Synergy

This work formalize and quantify data synergy in language model pretraining by leveraging observational variation across open-weight LLMs with diverse pretraining mixtures and estimating both direct domain-to-benchmark synergy and second-order domain-domain synergy (capabilities that require co-occurrence of multiple domains).