Clean: Second-order LLM Training at Linear Memory Cost via Nystr\"om Sketching
Training large language models (LLMs) entails a fundamental trade-off: memory-efficient optimizers such as Adam discard cross-parameter curvature, whereas full-curvature methods such as SOAP can accelerate convergence at prohibitive memory costs. We introduce Clean, a memory-efficient and full-curvature optimizer desig...
Beheshteh T. Rakhshan, S. Rajabi, Maziar Sargordi Shikai Fang et al.
· 0 citations