Stochastic ADMM with Balanced Augmented Lagrangian Method for Nonconvex and Nonsmooth Finite-Sum Optimization

Nonconvex Composite Functional Constraints via First-Order Augmented Lagrangian Methods under Local Regularity

We study nonasymptotic convergence of primal-dual methods for a class of nonconvex constrained optimization problems with a convex-composite structure. In this class, both the objective and the functional inequality constraints are given by convex Lipschitz outer functions composed with smooth nonlinear inner mappings. The analysis is complicated by constraint violation in a nonconvex functional inequality system and by the lack of an a priori bound on the multipliers. To address these issues, we restrict the dual variable to an auxiliary compact set and analyze a smoothed prox-linear augmented Lagrangian method through a nonsmooth nonconvex-concave minimax reformulation. The main contribution is a finite-time mechanism for converting stationarity of the truncated minimax problem into a KKT certificate for the original constrained problem. We show that, for a sufficiently large penalty parameter, all but a controlled number of iterates enter a near-feasible region. On this region, a local conic regularity condition uniformly bounds the associated prox-linear multipliers and thereby makes the artificial dual truncation inactive at the selected iterates. Building on this mechanism, we establish explicit convergence rates for the proposed method in terms of the KKT residual. With dual regularization, a global dual error bound together with a bias-balancing argument gives an $O(K^{-1/3})$ rate. In the unregularized case, under additional local structural assumptions including piecewise linearity of the outer functions, a local dual error bound yields the sharper $O(K^{-1/2})$ rate.

A proximal subgradient method for nonconvex stochastic optimization under the Kurdyka-{\L}ojasiewicz condition

This work introduces a proximal stochastic subgradient method for minimizing the sum of an expected cost, whose integrand is potentially nonsmooth and nonconvex, and a lower semicontinuous, prox-bounded function. We target a broad class of integrands obeying a nonsmooth, localized variant of the descent lemma in the decision variable, a structural assumption that simultaneously covers smooth losses with Lipschitz gradient and differences of such losses with convex functions. At each iteration the expected cost is replaced by a sample average that is progressively refined, and the proximal-subgradient stepsize is selected by an Armijo-type line search enforcing a sufficient-decrease property up to stochastic errors induced by the sample-based approximation. This framework accommodates substantially more general problem formulations than existing methods, in particular, it requires neither (weak) convexity of the regularizer nor a uniform bound on the variance of the stochastic oracle, and our analysis yields convergence guarantees that are new even in the smooth setting. Specifically, we establish almost sure convergence of the sequence of function values and stationarity of every accumulation point of the trajectories under the relaxed requirement that the sample-size sequence be merely nondecreasing and unbounded, with no prescribed growth rate. Leveraging the Kurdyka-Lojasiewicz (KL) property, we further upgrade this subsequential guarantee to convergence of the whole trajectory to a single stationary point. Finally, for exponential-type KL desingularizing functions and polynomially growing sample sizes, we derive explicit polynomial convergence rates, up to a logarithmic factor, for both the function values and the iterates.

The Variance Reduced Douglas-Rachford Splitting Method for Large-Scale Optimization Problems

An augmented Lagrangian method with exact multipliers for non-separable composite $\ell_0$-$\ell_2$ regularization

This paper studies a non-separable composite $\ell_0$-$\ell_2$ regularization model that simultaneously enforces sparsity and smoothness for inverse problems. The $\ell_0$ norm induces inherent nonconvexity and nonsmoothness, while linear transformations further introduce nonseparability, making the problem computationally challenging to solve. The existing inexact augmented Lagrangian method suffers from high computational complexity and unstable convergence. To overcome these difficulties, we develop two novel augmented Lagrangian algorithms with exact multipliers, designed respectively for the full row-rank case and the general matrix case, where all subproblems are globally optimized via closed-form solutions. Furthermore, we prove linear convergence of the proposed method when the transformation matrix is full row rank. In the general setting, all accumulation points of the generated sequence are KKT points for the original problem. Numerical experiments on synthetic data, trend filtering, and image smoothing demonstrate the superior efficiency and accuracy of the proposed methods over the existing method, confirming our theoretical analysis.

A Fixed-Penalty Linearized Augmented Lagrangian Method with Classical Multiplier Updates

Augmented Lagrangian methods are effective for nonlinear equality-constrained optimization, but solving their nonlinear primal subproblems can be expensive. For smooth nonconvex problems with deterministic or stochastic objectives, we propose a nonlinear-residual linearized augmented Lagrangian method (NR-LALM) that replaces this subproblem by a regularized Gauss-Newton-type step while retaining the classical multiplier update based on the nonlinear constraint residual. The resulting step is computed from one symmetric positive-definite linear system, but the mismatch between the linearized primal model and the nonlinear-residual update produces a quadratic constraint-linearization error in the multiplier identity. We show that this error can be controlled under local regularity; multiplier boundedness and trajectory localization are derived rather than assumed. With fixed, accuracy-independent parameters, deterministic NR-LALM finds an $\varepsilon$-approximate Karush-Kuhn-Tucker (KKT) pair in $O(\varepsilon^{-2})$ iterations and first-order oracle evaluations. For stochastic objectives, a projected stochastic path-integrated differential estimator with safeguarded restarts requires, in expectation, $O(\varepsilon^{-3})$ stochastic-gradient evaluations and $O(\varepsilon^{-2})$ constraint and Jacobian evaluations. Compactness and a Kurdyka-Lojasiewicz condition further yield finite-length convergence of the deterministic primal-dual sequence. An optional minimum-norm second-order correction reduces the constraint-linearization error from second to fourth order without changing the complexity orders. All theoretical results are formalized in Lean 4. Numerical experiments confirm the predicted error orders and show favorable performance on high-dimensional deterministic and stochastic problems.

The Augmented Lagrangian Methods: Overview and Recent Advances

Abstract. Large-scale constrained optimization is pivotal in modern scientific, engineering, and industrial computation, often involving complex systems with numerous variables and constraints. This paper provides a unified and comprehensive perspective on constructing augmented Lagrangian functions (based on the Hestenes–Powell–Rockafellar augmented Lagrangian) for various optimization problems, including nonlinear programming and convex and nonconvex composite programming. We present the augmented Lagrangian method (ALM), covering its theoretical foundations in both convex and nonconvex cases, and discuss several successful examples and applications. Recent advancements have extended the ALM’s capabilities to handle nonconvex constraints and ensure global convergence to first- and second-order stationary points. For nonsmooth convex problems, the ALM utilizes proximal operations, preserving desirable properties such as locally linear convergence rates. Furthermore, recent progress has refined the complexity analysis of the ALM and tackled challenging integer programming instances. This review aims to offer a thorough understanding of the ALM’s benefits and limitations, exploring different ALM variants designed to enhance convergence and computational performance. We also illustrate effective algorithms for ALM subproblems across different types of optimization problems and highlight practical implementations in several fields.