The theoretical results show that GN's optimality for linear regression no longer holds under a poorly chosen basis under both population and stochastic updates, or a move from linear regression to a non-convex variant of logistic regression, and Adam-style methods offer genuine advantages over curvature-inspired preco...
It is found that training on code-switched data aligns the representations of parallel text, particularly across different scripts, and that this alignment persists through training on monolingual documents.
Dries Rooryck, Alex Cai, Yonatan Belinkov et al.· 0 citations
This work formalize and quantify data synergy in language model pretraining by leveraging observational variation across open-weight LLMs with diverse pretraining mixtures and estimating both direct domain-to-benchmark synergy and second-order domain-domain synergy (capabilities that require co-occurrence of multiple d...
K. Hamidieh, Lester Mackey, David Alvarez-Melis· arXiv.org· 4 citations
ADAPT---Amortized Distillation Across Post-Trained LLMs is introduced, a framework for amortizing distillation across both axes of a model family: size and post-training variant, producing models for interpolated sizes across post-trained variants with a single distillation run.
Yan Zhou, Sara Kangaslahti, Jonathan Geuter et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
