A model-based minimization method is developed under a relative Lipschitz condition and a framework with convergence guarantees is extended to the setting where the distance generating function and its gradient are accessible only through a stochastic oracle.
Abstract
Composite optimization plays a central role in modern machine learning and signal processing, as it offers a natural balance between data fidelity and structural properties. In this paper, we study composite optimization in the setting where both components are nonsmooth and nonconvex. We start with a deterministic Bregman proximal subgradient method that converges under subgradient upper-bound conditions. This approach relaxes the standard requirement on the convexity of the regularization term, thus accommodating a broader range of applications. To extend this to the stochastic regime, we develop a model-based minimization method under a relative Lipschitz condition and establish a convergence rate of $\mathcal{O}(\varepsilon^{-4})$. We also extend the framework with convergence guarantees to the setting where the distance generating function and its gradient are accessible only through a stochastic oracle.
We investigate the optimization problem of minimizing a nonsmooth function that satisfies a nonsmooth version of the descent lemma over a nonempty and closed but not necessarily convex set. The objective function belongs to the class of upper-$\mathcal{C}^2$ functions, whereas the constraints may promote a sparse or low-rank structure. We propose a projected subgradient method with two different globalization strategies: (a) a nonmonotone linesearch and, under additional assumptions, (b) an auto-conditioned method, where the stepsize is given by a formula depending on data from past iterations. We show that both methods converge to solutions that satisfy a stronger stationarity concept than one would expect from the subdifferential sum-rule, which is particularly important since the optimization problems of interest are inherently nonconvex. Finally, we present promising numerical results when applying the algorithm to an MPEC-style problem as well as the matrix optimization problems MAXCUT and Robust PCA.
Christian Kanzow, Jannis Krüger, Leo Lehmann· 0 citations
This work introduces a proximal stochastic subgradient method for minimizing the sum of an expected cost, whose integrand is potentially nonsmooth and nonconvex, and a lower semicontinuous, prox-bounded function. We target a broad class of integrands obeying a nonsmooth, localized variant of the descent lemma in the decision variable, a structural assumption that simultaneously covers smooth losses with Lipschitz gradient and differences of such losses with convex functions. At each iteration the expected cost is replaced by a sample average that is progressively refined, and the proximal-subgradient stepsize is selected by an Armijo-type line search enforcing a sufficient-decrease property up to stochastic errors induced by the sample-based approximation. This framework accommodates substantially more general problem formulations than existing methods, in particular, it requires neither (weak) convexity of the regularizer nor a uniform bound on the variance of the stochastic oracle, and our analysis yields convergence guarantees that are new even in the smooth setting. Specifically, we establish almost sure convergence of the sequence of function values and stationarity of every accumulation point of the trajectories under the relaxed requirement that the sample-size sequence be merely nondecreasing and unbounded, with no prescribed growth rate. Leveraging the Kurdyka-Lojasiewicz (KL) property, we further upgrade this subsequential guarantee to convergence of the whole trajectory to a single stationary point. Finally, for exponential-type KL desingularizing functions and polynomially growing sample sizes, we derive explicit polynomial convergence rates, up to a logarithmic factor, for both the function values and the iterates.
Felipe Atenas, Alejandro Jofré, Pedro Pérez-Aros et al.· 0 citations
We study policy optimization for discrete-time robust $\mathcal{H}_\infty$ control with static output-feedback, and present the first feasibility-preserving algorithm with a deterministic, non-asymptotic complexity guarantee. This problem naturally leads to a nonsmooth and nonconvex optimization over the set of stabilizing feedback gains. We first establish several structural properties of the $\mathcal{H}_\infty$ cost. In particular, we show that the cost is weakly convex on every convex subset of a sublevel set. For the state-feedback case, we further establish a weak Polyak--{\L}ojasiewicz inequality, which ensures that every stationary point is globally optimal. Building on these properties, we develop a proximal bundle method for $\mathcal{H}_\infty$ policy optimization. The proposed method can be viewed as an implementable approximation of the proximal point method and uses only function value and subgradient information. We show that all iterates remain stabilizing and establish a deterministic non-asymptotic complexity bound of $\mathcal{O}(\max\{\eta^{-4},\epsilon^{-2}\})$ for finding an $(\eta,\epsilon)$-stationary point. Numerical experiments illustrate our theoretical results.
We study Langevin-based methods for non-convex optimization under smoothness and dissipativity assumptions. Our focus is on obtaining non-asymptotic bounds for the expected excess risk rather than sampling guarantees for the full target distribution. The key ingredient of our analysis is a direct passage from relative entropy to objective-value error, based on a weighted Csisz\'ar--Kullback--Pinsker inequality and exponential-moment estimates. This avoids intermediate Wasserstein bounds and yields sharper dependence on the Log-Sobolev constant, a quantity that may scale exponentially with the inverse temperature and the dimension in non-convex problems. We first analyze the Unadjusted Langevin Algorithm with exact gradients and derive explicit bounds on $\mathbb{E}[F(x_k)]-\min F$ in terms of the inverse temperature, dimension, stepsize, smoothness and dissipativity parameters, and the Log-Sobolev constant. We then extend the result to an inexact-gradient version of ULA, allowing for biased and stochastic gradient surrogates whose mean-square error grows at most quadratically in the state. This framework covers stochastic gradients and zeroth-order estimators based only on function evaluations. In particular, we show that both Gaussian and spherical finite-difference estimators fit into the inexact-ULA theory and obtain explicit function-evaluation complexity bounds for zeroth-order Langevin optimization. To the best of our knowledge, these are the first non-asymptotic global non-convex optimization complexity bounds for zeroth-order ULA. We also provide numerical experiments illustrating the behavior of the proposed zeroth-order Langevin schemes.
E. Naldi, Marco Rando, Lorenzo Rosasco et al.· 0 citations
In a real Hilbert space, we study a bilevel optimization problem that consists in minimizing an outer convex function over the zero set of a maximally monotone operator. In the smooth setting, where the outer objective is convex and Fr\'echet differentiable and the inner operator is single-valued, continuous and monotone, we associate with the problem a first-order dynamical system that can be viewed as a monotone flow applied to a dynamically regularized operator. Under suitable geometric conditions on the inner problem --- either a weak Attouch-Czarnecki-type integrability condition or the stronger assumption of sharpness --- we establish last-iterate convergence rates for both the outer and inner residuals, together with weak convergence of the trajectories to optimal solutions of the bilevel problem. In the smooth+nonsmooth setting, we enrich the outer objective with a proper, convex, and lower semicontinuous function, while the inner operator is augmented by the subdifferential of a function with the same properties. We propose a regularized proximal-extragradient algorithm in which both the forward and backward steps are performed with respect to dynamically regularized operators and functions, respectively. Under geometric assumptions on the inner problem analogous to those in the smooth setting, we establish last-iterate convergence rates for both the outer and inner residuals, together with weak convergence of the iterates to optimal solutions of the bilevel problem.
We study a class of weakly convex optimization problems in which the objective is the sum of a smooth convex term and a weakly convex term that may be nonsmooth. To exploit this structure, we develop a splitting technique based on the alternating direction method of multipliers (ADMM), which decouples the minimization of the two components into tractable subproblems. Because the update associated with the smooth term may require an inner iterative solver, we further linearize this term, yielding a linearized ADMM (LADMM) scheme with an inexpensive one-step update. Under mild conditions, we establish the subsequence convergence of both ADMM and LADMM methods to directional stationary solutions, which are equivalent to critical points and Clarke stationary solutions for our weakly convex problem. Numerical experiments on two low-dimensional test functions and a high-dimensional logarithmic regularized logistic regression model demonstrate that the proposed approaches are computationally efficient and produce solutions of comparable quality to baseline methods.
Sheng-Han Mei, Cheng-Yu Ke, Yifei Lou et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
