Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization
Borders reveal a noise--drift tradeoff: in noise-dominated regimes, first-moment averaging and adaptive preconditioning can yield favorable upper guarantees, whereas in drift-dominated regimes, stale first-moment information and preconditioning can enlarge Adam's tracking guarantee, potentially allowing vanilla SGD to...