This paper establishes last-iterate convergence of stochastic first-order methods for constrained smooth convex--concave minimax optimization under the standard bounded-variance stochastic oracle and establishes two types of convergence guarantees.
Abstract
In this paper, we study last-iterate convergence of stochastic first-order methods for constrained smooth convex--concave minimax optimization under the standard bounded-variance stochastic oracle. A fundamental challenge is that the last iterates of vanilla stochastic extragradient (S-EG) and stochastic optimistic gradient descent--ascent (S-OGDA) may fail to converge in the presence of stochastic gradient noise, even for simple bilinear problems. To overcome this difficulty, we introduce a simple perturbation framework that regularizes the original convex--concave problem into a strongly convex--strongly concave one. Applying S-EG and S-OGDA to the perturbed problem yields two simple single-loop methods, referred to as perturbed S-EG (PS-EG) and perturbed S-OGDA (PS-OGDA). We establish last-iterate convergence by first deriving convergence in terms of the squared distance to the saddle point of the perturbed problem and then translating this estimate into guarantees for the restricted primal--dual gap. Based on this framework, we establish two types of convergence guarantees. When the optimization horizon is known \emph{a priori}, both PS-EG and PS-OGDA achieve an $\mathcal{O}(T^{-1/4})$ last-iterate convergence rate for the restricted primal--dual gap, which coincides with the standard primal--dual gap on compact feasible domains. When the optimization horizon is unknown, we develop an anytime variant based on diminishing perturbations and diminishing stepsizes. For general closed convex feasible sets, both PS-EG and PS-OGDA achieve an $\mathcal{O}(T^{-1/5})$ last-iterate convergence rate for the restricted primal--dual gap. Furthermore, in the unconstrained setting, PS-EG admits a sharper $\mathcal{O}(T^{-1/4})$ anytime convergence rate in terms of the gradient norm.
This work proposes a novel single-loop algorithm based on a constrained reformulation in which lower-level stationarity is imposed as a constraint, and constructs a regularized Lagrangian by introducing a quadratic regularizer and restricting the dual variable to a bounded domain.
A new convergence rate for SMG in terms of the squared Pareto-stationarity (PS) measure is established, to exploit the Lipschitz continuity of the PS measure, defined by the norm of the multi-gradient descent algorithm (MGDA) direction, rather than the $(1/2)-H\"older continuity of the MGDA direction.
This paper considers the design of optimal fixed-step first-order methods for high-dimensional minimization of $L$-smooth convex functions. For optimizing worst-case performance measured via suboptimality of the final function value (relative to the initial squared distance to a minimizer), we provide an algebraic proof of the optimality of the optimized gradient method (OGM) and establish its uniqueness among all fixed-step first-order methods. For the alternative measure of final squared gradient norm (relative to initial suboptimality), we prove the OGM-G method is optimal and uniquely so among fixed-step first-order methods. Finally, for the setting measuring the final squared gradient norm (relative to the initial squared distance to a minimizer), we show the recently proposed Lemniscate method is optimal and uniquely so. Our proofs rely on algebraic reductions for lower bound arguments rather than traditional information-theoretic bounds, which were previously only able to establish OGM's optimality but not uniqueness.
Benjamin Grimmer, Sunghyeon Jo, Chanwoo Park· 0 citations
In smooth convex optimization, the gradient norm is a directly observable measure of stationarity. Accelerating a first-order method that minimizes the gradient norm is known to be more delicate than accelerating the minimization of function values. Optimal accelerated methods such as OGM-G (Kim&Fessler, 2021) are known to exist for any prescribed finite horizon, but their coefficients depend explicitly on the length of that horizon, i.e. the number of iterations. We ask what kind of acceleration is feasible when the stopping horizon is unknown to the method, i.e. for horizon-independent methods. Diakonikolas&Wang (2022) conjectured that an $\Omega(N^{-1})$ lower bound on the squared gradient norm holds at every horizon $N$ for any nonadaptive, horizon-independent linear-span first-order method. We disprove this pointwise conjecture by exhibiting a method that achieves near-$N^{-2}$ last-iterate guarantees on a density-one set of horizons. We show instead that an $\Omega(N^{-1})$ lower bound must hold for infinitely many horizons. More precisely, if $\mathcal G_N(\mathcal A)$ denotes the bound on the squared gradient norm after $N$ iterations for a method $\mathcal A$, we prove that $\limsup_{N\to\infty}N \mathcal G_N(\mathcal A) \ge 1/2$ for any method $\mathcal A$. This bound is sharp: the constant $1/2$ is exactly attained by the horizon-independent gradient-descent schedule of Rotaru et al. (2026). In addition, we show that the above two extreme behaviors cannot be achieved by the same method: any method $\mathcal A$ with an $o(N^{-1})$ guarantee on a subsequence of iterates must satisfy $\limsup_N N \mathcal G_N(\mathcal A)=\infty$. In contrast, best-so-far output admits a uniform $O(N^{-2})$ guarantee.
In this paper, we study a Tikhonov-regularized mixed-order primal--dual dynamical system with implicit Hessian damping for linearly constrained convex optimization problems in finite-dimensional Euclidean spaces, where the primal equation is second order and incorporates the viscous damping term \(\delta\sqrt{\varepsilon(t)}\,\dot x(t)\), whereas the multiplier equation remains first order. By constructing a new class of energy functions, for a general Tikhonov regularization coefficient \(\varepsilon(t)\), we prove the strong convergence of the primal trajectory and derive fast convergence rates under the same parameter assumptions, without imposing any eventual inside/outside-ball condition. More precisely, the primal trajectory converges to the minimum-norm solution, and the multiplier converges to a compatible KKT multiplier, while the convergence rates of the Lagrangian gap, feasibility violation, and objective residual are \(o(\varepsilon(t))\), and the convergence rate of the velocity norm is \(o(\sqrt{\varepsilon(t)})\). For the critical case \(\varepsilon(t)=c/t^2\), in which the damping coefficient \(\delta\sqrt{\varepsilon(t)}\) reduces to \(\delta\sqrt{c}/t\), we establish the sharper convergence rates \(o(t^{-2})\) for the Lagrangian gap, feasibility violation, and objective residual, together with \(o(t^{-1})\) for the velocity norm, which improve the corresponding \(O(t^{-2})\) and \(O(t^{-1})\) decay estimates obtained in the related literature. Most importantly, when the proposed dynamical system is specialized to the finite-dimensional unconstrained setting, our analysis answers the open question on strong convergence in this critical regime posed by Attouch and L\'aszl\'o [Math. Methods Oper. Res., 99 (2024), pp.~307--347].
We consider stochastic multi-objective optimization over a nonempty closed convex set, where every objective is an expectation and only sample-gradient information is available. We develop a line-search-free and function-value-free adaptive projected-gradient algorithm for the sample-average approximation (SAA) problem. Each iteration computes a feasible regularized multi-gradient step and updates the regularization parameter from the projected step length. A normal-cone-based certificate yields descent estimates and an explicit complexity bound for the Pareto-stationarity residual of the SAA problem. The consistency of SAA gradients then transfers vanishing SAA residuals to Pareto stationarity for the population problem, while an additional concentration argument gives a finite-sample residual bound on compact sets. Experiments on synthetic problems, classification, portfolio selection, multi-task learning, and robot control illustrate the practical performance of our algorithm.
Yi-Yang Li, Lei Wang, Xiaojun Chen· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
