#machine learning
Feb 2026
MSign: An Optimizer Preventing Training Instability in Large Language Models via Stable Rank Restoration
This work proposes MSign, a new optimizer that periodically applies matrix sign operations to restore stable rank in a 5M-parameter NanoGPT model, and proves theoretically that these two conditions jointly cause exponential gradient norm growth with network depth.
Lianhai Ren, Yucheng Ding, Xiao Liu et al.
· arXiv.org · 1 citation