This work proposes a time-smoothed proximal linear algorithm and a local-regret measure based on a proximal residual mapping that is a proper stationarity measure for the original problem: its fixed-point condition implies first-order stationarity.
Abstract
We study online optimization for a broad class of structured non-convex non-smooth problems where each loss is a composition of a difference-of-convex function with a smooth mapping, and the feasible region is defined by constraint functions of the same kind. We propose a time-smoothed proximal linear algorithm and a local-regret measure based on a proximal residual mapping. We show that this residual is a proper stationarity measure for the original problem: its fixed-point condition implies first-order stationarity. Our analysis relies on a tangent-cone characterization for a feasible region described by composite difference-of-convex constraints, which is of independent interest and allows each update to be computed via a convex optimization oracle, despite the non-convexity of the problem. We establish a local-regret bound and a bound on the total number of inner convex subproblems. We also derive an error bound connecting the proximal residual to the distance to stationarity, providing a quantitative certificate of approximate stationarity.
We propose MELMO (Moreau Envelope Smoothing with Linear Minimization Oracles), an algorithm for composite optimization problems of the form min x f (x) + g(T x), where f is smooth and g may be non-smooth. The method leverages the Moreau envelope to smooth the non-smooth component while adapting to problem geometry through linear minimization oracles. Assuming g is $\rho$-weakly convex, we establish a family of convergence bounds parameterized by the step-size and smoothing schedules, thereby making explicit the trade-off between optimizing the smoothed objective and recovering stationarity for the original composite problem. In particular, one regime yields O(k -1/4 ) rates for both the smoothed-gradient norm and a composite stationarity proxy, while another yields O(k -1/3 ) for the smoothed-gradient norm together with O(k -1/4 ) for the composite proxy. We also establish a K-horizon-dependent convergence rate that yields O(K -1/3 ) for the composite proxy. Empirically, MELMO is competitive with variable smoothing and subgradient baselines on sparse low-rank matrix factorization and image denoising.
This work presents two different types of proximal gradient methods, with line search and without line search, for solving unconstrained set-valued optimization problems under the lower set-less ordering relation induced by a solid cone that is convex, pointed, and closed. The objective mapping of the problem involves finitely many functions, with each one being the sum of a continuously differentiable function and a convex function that is proper and closed. We present an approach to characterize weakly minimal points of the problem with the help of weakly efficient points of a family of vector optimization problems. Thereafter, we establish a stationarity condition along with its connection with weakly minimal points of the problem under study. Based on the stationary condition, the concept of a descent direction at a non-stationary point is discussed. In view of the line search-based method, we formulate an Armijo-type line search condition and establish the existence of such a step-size. For the proposed methods, global convergence is established under mild assumptions. The convergence analysis of the proximal gradient method with line search provides a theoretical advancement over the convergence results previously established for the steepest descent method in set-valued optimization problems. In addition, we analyze the computational complexity of the proposed methods and show that both methods achieve a convergence rate of $\mathcal{O}(1/\sqrt{k})$. Numerical results are reported to test the performance of the methods in practice.
Ravi Raushan, Debdas Ghosh, Anshika et al.· 0 citations
Minimizing gradients of a convex function is an important problem across optimization and learning tasks. The gradient provides a directly computable certificate of approximate stationarity, and its minimization usually implies stronger results than those for minimization of function values. In this work, we study gradient-norm minimization for convex functions that are $(L,\kappa)$-H\"older smooth with respect to the $\ell_p$-norms, $p \geq 1$. We develop algorithms that achieve near-optimal gradient-oracle complexity for this problem. In the smooth case, our results resolve the previously open setting $p>2$. For H\"older-smooth objectives, we close the complexity gap throughout the full $p$-range, including to the best of our knowledge, a gap in the Euclidean case. We provide two families of algorithms: the first one comes with a simple iteration and generalizes a phenomenon known as mirror duality, exploiting dual behaviours of algorithms with errors and inexact computations. The second makes use of accumulating regularizers centered at different approximate solutions, which we sequentially minimize in order to provide our near-optimal rates.
Nico Pelleriti, Maryam Shiran, David Martínez-Rubio et al.· 0 citations
In a real Hilbert space, we study a bilevel optimization problem that consists in minimizing an outer convex function over the zero set of a maximally monotone operator. In the smooth setting, where the outer objective is convex and Fr\'echet differentiable and the inner operator is single-valued, continuous and monotone, we associate with the problem a first-order dynamical system that can be viewed as a monotone flow applied to a dynamically regularized operator. Under suitable geometric conditions on the inner problem --- either a weak Attouch-Czarnecki-type integrability condition or the stronger assumption of sharpness --- we establish last-iterate convergence rates for both the outer and inner residuals, together with weak convergence of the trajectories to optimal solutions of the bilevel problem. In the smooth+nonsmooth setting, we enrich the outer objective with a proper, convex, and lower semicontinuous function, while the inner operator is augmented by the subdifferential of a function with the same properties. We propose a regularized proximal-extragradient algorithm in which both the forward and backward steps are performed with respect to dynamically regularized operators and functions, respectively. Under geometric assumptions on the inner problem analogous to those in the smooth setting, we establish last-iterate convergence rates for both the outer and inner residuals, together with weak convergence of the iterates to optimal solutions of the bilevel problem.
We investigate the optimization problem of minimizing a nonsmooth function that satisfies a nonsmooth version of the descent lemma over a nonempty and closed but not necessarily convex set. The objective function belongs to the class of upper-$\mathcal{C}^2$ functions, whereas the constraints may promote a sparse or low-rank structure. We propose a projected subgradient method with two different globalization strategies: (a) a nonmonotone linesearch and, under additional assumptions, (b) an auto-conditioned method, where the stepsize is given by a formula depending on data from past iterations. We show that both methods converge to solutions that satisfy a stronger stationarity concept than one would expect from the subdifferential sum-rule, which is particularly important since the optimization problems of interest are inherently nonconvex. Finally, we present promising numerical results when applying the algorithm to an MPEC-style problem as well as the matrix optimization problems MAXCUT and Robust PCA.
Christian Kanzow, Jannis Krüger, Leo Lehmann· 0 citations
This article derives a projected gradient flow characterized by tangent and normal cone decompositions, which capture the local geometry of the constraint set and shows that constraint geometry continuously filters gradient directions along the optimization trajectory, leading to a trajectory-dependent implicit regularization effect without modifying the objective function.
Yan-Jun Yan· Big Data· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
