Which Geometry on Which Layer? Measured Sign-Spectral Assignment in Mixed-Optimizer Training
Modern neural-network training uses mixed recipes, assigning AdamW-like or sign-based updates to embedding and output layers and spectral updates to hidden matrices. These assignments are usually fixed by layer type, without a clear criterion for selecting the appropriate geometry. We turn sign-versus-spectral assignme...
Hiroki Naganuma, Laura Gomezjurado, Mahdi Ghaznavi et al.
· 0 citations