Gao-Xiang Tang

Beyond Quadratic Loss: The Stability Phase Diagram of Adam

This work connects Adam loss spikes to both the mismatch between momentum timescales and finite-scale superquadratic loss geometry beyond the Hessian, and shows that confident cross-entropy losses develop a core--wall landscape comprising a narrow quadratic core followed by a steep wall, which produces effective superq...