Liang Hou

ORCA: The Annealed Spectral Conditioning Optimizer for Faster, Better LLM Training

Modern LLM optimizers such as Muon often produce weight matrices with higher effective rank than Adam, yet further spectral control has delivered only modest gains. We identify a tension behind this result: concentrated spectra can suppress gradient directions in coupled weight matrices and slow optimization, while con...