The Complexity of Finding Stationary Points in Nonsmooth Nonconvex Optimization

Abstract

We prove that first-order algorithms require $\Omega(\delta^{-1}\epsilon^{-3})$ gradient queries (in the worst case) to find a $(\delta,\epsilon)$-Goldstein stationary point of a Lipschitz function, at which there is a convex combination of gradients within distance $\delta$ whose norm is at most $\epsilon$. This lower bound is tight, matching known algorithms up to absolute constants, therefore resolving the complexity of convergence to stationarity in nonsmooth nonconvex optimization. We further prove a tight lower bound of $\Omega(\lambda^{1/2}\epsilon^{-7/2})$ for finding points satisfying the recently proposed relaxed notion of $(\lambda,\epsilon)$-stationarity, which allows combining further-away gradients. Our results reveal that convergence rates to nonsmooth stationarity are not affected by gradient stochasticity, in sharp contrast to smooth optimization.