Jul 2026
SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales
This work adapt and enhance preconditioned gradient methods to overcome the practical challenges of large-scale LLM pretraining and proposes algorithmic modifications including per-step QR orthogonalization and improved preconditioning strategies to enable stable training in these regimes.
Mikail Khona, Aditya Vavre, Boxiang Wang et al.
· arXiv.org · 0 citations