Adam or Gauss-Newton? A Comparative Study In Terms of Basis Alignment and SGD Noise
TL;DR
The theoretical results show that GN's optimality for linear regression no longer holds under a poorly chosen basis under both population and stochastic updates, or a move from linear regression to a non-convex variant of logistic regression, and Adam-style methods offer genuine advantages over curvature-inspired preconditioning.
Abstract
Preconditioned methods are central to deep learning optimization. Predominant approaches include computationally light diagonal preconditioners such as Adam, which rely on gradient statistics, and second-order methods such as Gauss-Newton (GN), which capture richer curvature information. Seeking the best of both worlds, we disentangle the preconditioner design space into several factors, separating the choice of diagonal scaling (Adam-style versus GN-style) from 1) the basis choices under which the diagonal scaling operates, and 2) the gradient noises from mini-batching. Our theoretical results show that GN's optimality for linear regression no longer holds under a poorly chosen basis under both population and stochastic updates, or a move from linear regression to a non-convex variant of logistic regression. In these settings, Adam-style methods offer genuine advantages over curvature-inspired preconditioning, rather than serving merely as a tractable proxy. Empirical results on synthetic problems and CIFAR-10 support these findings.