Orthogonal Witness Control for Muon Optimization via Sigmoid Spectral Reshaping
A matrix-valued optimizer that preserves the singular subspaces of the gradient while applying a bounded, monotone sigmoid transformation to its singular values is introduced, and the effectiveness and robustness of Soren against established optimizers are demonstrated.
Dat Phi Van, Minh Vu Ngo, Tuc Nguyen et al.
· 0 citations