Huan Li

On the O(√d/K1/4) Convergence Rate of AdamW Measured by ℓ1 Norm

Experimental results on real-world deep learning tasks and theoretical results support that the convergence rate of AdamW can be considered to be analogous to the optimal $\frac{1}{K}\sum_{k=1}^KE\left[||\nabla f(x^k)||_2\right]\leq O(\frac{C}{K^{1/4}})$ convergence rate of SGD in the ideal case.