Variance-Reduced Conditional Gradient Methods under Markovian Sampling for Nonconvex Composite Optimization

Improved Convergence Rates for Stochastic Multi-Gradient Descent which Close the Gap: A Proof by AI

A new convergence rate for SMG in terms of the squared Pareto-stationarity (PS) measure is established, to exploit the Lipschitz continuity of the PS measure, defined by the norm of the multi-gradient descent algorithm (MGDA) direction, rather than the $(1/2)-H\"older continuity of the MGDA direction.

Generalization Error Estimation for Primal--Dual Algorithms in Non-Smooth Regression

This paper studies trajectory-wise estimation of generalization error for primal--dual algorithms in non-smooth regression. Motivating examples include \(\ell_1\)-penalized least absolute deviations regression and square-root Lasso regression, where the data-fitting loss is non-differentiable and existing risk estimators for gradient-type optimization paths do not apply directly. We develop a general recursive framework that includes the Chambolle--Pock algorithm and related primal--dual splitting methods. We estimate risk by correcting each in-sample fitted value with a weighted combination of past dual iterates. The ideal weights are Stein derivative contractions and depend on the design covariance. We construct replacement weights from observable derivative contractions of the fitted-signal trajectory, yielding a covariance-free, data-driven correction. For high-dimensional Gaussian designs and fixed finite iteration horizon, we prove finite-sample guarantees for both estimators. For square-root ridge, we further establish a matched-Gaussian universality result beyond Gaussian designs. Numerical experiments show that the proposed estimators accurately track the out-of-sample risk along finite optimization paths.

Hessian-Inverse-Free Stochastic Gradient Method for Stochastic Multi-Objective Bilevel Optimization

Stochastic Nonconvex Bilevel Optimization: Improved Rates Without Rare-Visit Assumption

We investigate stochastic simple bilevel optimization with smooth and possibly nonconvex upper- and lower-level objectives. Existing stochastic extensions of dynamic barrier gradient descent (DBGD) either obtain fast convergence under an unverifiable trajectory-dependent ``rare-visit''assumption, or remove this assumption at a substantially higher oracle cost. We show that a simple denominator-only regularization of the DBGD multiplier eliminates the need for such an assumption while preserving fast convergence rates. Specifically, our method achieves $(\varepsilon, \varepsilon)$-stationarity in $O(\varepsilon^{-2})$ iterations using $O(\varepsilon^{-4})$ upper-level and $O(\varepsilon^{-7})$ lower-level stochastic gradients, which improves upon the best assumption-free complexities. We additionally derive anytime parameter schedules.

Stability of Finite-Batch Particle Mean-Field Variational Inference Beyond Strong Convexity

We study the implementable finite-batch particle algorithm for mean-field variational inference as a fully discrete stochastic approximation of the projected Wasserstein dynamics. The target potential is globally smooth but need not be strongly convex. The departure from contractivity is quantified by the curvature defect \[ \mathfrak d_\alpha(x,y) = \bigl[\alpha\|x-y\|^2- \langle\nabla V(x)-\nabla V(y),x-y\rangle\bigr]_+, \] which is the additive loss in the one-step Euler contraction estimate. We prove a non-asymptotic Wasserstein stability bound that separates initialization, product-empirical approximation, finite-batch drift error, time discretization, and the defects accumulated along the coupled trajectories. Under the uniform bound $\mathfrak d_\alpha\leq\beta$, the particle iterates remain within $O(\sqrt{\beta/\alpha})$ of any MFVI minimizer, up to explicit errors in the particle number, batch size, and step size. The proof uses a stationary comparison array whose population law is an MFVI minimizer but whose particle-level law is a random product empirical measure, and it controls the resulting projected-drift discrepancy explicitly. We also give coordinatewise defect estimates and structural conditions for dimension-independent projected-drift sensitivity, construct an arbitrary-dimensional smooth nonconvex benchmark with a closed-form MFVI minimizer, and explain why polynomially growing drifts require a modification of the untamed explicit scheme.

A proximal subgradient method for nonconvex stochastic optimization under the Kurdyka-{\L}ojasiewicz condition

This work introduces a proximal stochastic subgradient method for minimizing the sum of an expected cost, whose integrand is potentially nonsmooth and nonconvex, and a lower semicontinuous, prox-bounded function. We target a broad class of integrands obeying a nonsmooth, localized variant of the descent lemma in the decision variable, a structural assumption that simultaneously covers smooth losses with Lipschitz gradient and differences of such losses with convex functions. At each iteration the expected cost is replaced by a sample average that is progressively refined, and the proximal-subgradient stepsize is selected by an Armijo-type line search enforcing a sufficient-decrease property up to stochastic errors induced by the sample-based approximation. This framework accommodates substantially more general problem formulations than existing methods, in particular, it requires neither (weak) convexity of the regularizer nor a uniform bound on the variance of the stochastic oracle, and our analysis yields convergence guarantees that are new even in the smooth setting. Specifically, we establish almost sure convergence of the sequence of function values and stationarity of every accumulation point of the trajectories under the relaxed requirement that the sample-size sequence be merely nondecreasing and unbounded, with no prescribed growth rate. Leveraging the Kurdyka-Lojasiewicz (KL) property, we further upgrade this subsequential guarantee to convergence of the whole trajectory to a single stationary point. Finally, for exponential-type KL desingularizing functions and polynomially growing sample sizes, we derive explicit polynomial convergence rates, up to a logarithmic factor, for both the function values and the iterates.