We study Langevin-based methods for non-convex optimization under smoothness and dissipativity assumptions. Our focus is on obtaining non-asymptotic bounds for the expected excess risk rather than sampling guarantees for the full target distribution. The key ingredient of our analysis is a direct passage from relative entropy to objective-value error, based on a weighted Csisz\'ar--Kullback--Pinsker inequality and exponential-moment estimates. This avoids intermediate Wasserstein bounds and yields sharper dependence on the Log-Sobolev constant, a quantity that may scale exponentially with the inverse temperature and the dimension in non-convex problems. We first analyze the Unadjusted Langevin Algorithm with exact gradients and derive explicit bounds on $\mathbb{E}[F(x_k)]-\min F$ in terms of the inverse temperature, dimension, stepsize, smoothness and dissipativity parameters, and the Log-Sobolev constant. We then extend the result to an inexact-gradient version of ULA, allowing for biased and stochastic gradient surrogates whose mean-square error grows at most quadratically in the state. This framework covers stochastic gradients and zeroth-order estimators based only on function evaluations. In particular, we show that both Gaussian and spherical finite-difference estimators fit into the inexact-ULA theory and obtain explicit function-evaluation complexity bounds for zeroth-order Langevin optimization. To the best of our knowledge, these are the first non-asymptotic global non-convex optimization complexity bounds for zeroth-order ULA. We also provide numerical experiments illustrating the behavior of the proposed zeroth-order Langevin schemes.
We study the computation of static mean-field equilibria on a compact state space by formulating the equilibrium condition as a variational inequality over probability measures. We propose an entropic variant of Korpelevich's extragradient algorithm---the Kullback--Leibler Mirror-Prox method---in which Euclidean projections are replaced by relative-entropy proximal steps. Each half-step is therefore an explicit exponential reweighting of the current measure, implemented on a finite state-space discretization. Under Lasry--Lions monotonicity and continuity assumptions, we prove convergence of mesh-refined ergodic averages and obtain finite-iteration Minty-residual and approximate-equilibrium bounds that jointly quantify iteration and discretization errors. Under strong monotonicity, we derive metric convergence rates for the last, best, and averaged iterates. We also develop a KL-type Tikhonov regularization that selects the equilibrium minimizing relative entropy with respect to a reference measure. The framework applies to potential and nonpotential cost operators and does not require differentiability or convexity of the cost in the individual state.
Erhan Bayraktar, Ibrahim Ekren, L. Vy et al.· 0 citations
Hessian-free high-resolution (HFHR) dynamics augments underdamped Langevin dynamics (ULD) with reversible position diffusion for sampling problems that arise in machine learning. We establish an explicit quantitative contraction rate for HFHR dynamics under a position Poincar\'e inequality, weighted Hessian and Laplacian bounds, and a compact Sobolev embedding, where the potential function is not necessarily convex. An adapted time-augmented Poincar\'e inequality yields an explicit rate that improves upon the contraction rate of the underdamped Langevin dynamics. We also give a weak-solution construction and a self-contained spectral proof of the divergence lemma underlying the argument. For HFHR Monte Carlo (HFHRMC) algorithm, which is based on a discretization scheme of HFHR dynamics, we use a path-space Girsanov argument to obtain a non-asymptotic convergence bound and an explicit iteration complexity in total variation distance. The bounds hold for every $\alpha\geq0$ and $\gamma>0$ and remain regular at the ULD endpoint. Optimizing the iteration complexity bound yields a positive, accuracy-dependent position-diffusion parameter at finite accuracy, while its leading high-accuracy order coincides with that of the optimized ULD endpoint. Our iteration complexity bound improves upon the existing work on HFHR algorithms. Numerical experiments including Bayesian learning problems on real data are provided to illustrate the effect of positive $\alpha$ and its benefit.
Wu-Jun Lv, Xiaoyu Wang, Yingli Wang et al.· 0 citations
We study fractional-Sobolev Tikhonov regularization for linear inverse problems on a bounded Lipschitz domain. The regularization penalty is generated by the restricted Dirichlet fractional Laplacian, and the associated variational problem is shown to admit a unique minimizer that depends Lipschitz continuously on the data. Identifying the positive self-adjoint operator $$A_s=I+(-\Delta)^s,\, D(A_s^{1/2})=H_0^s(\Omega),$$ we transform the problem isometrically into a classical Hilbert-space Tikhonov problem with observation operator $B=KA_s^{-1/2}$. This yields explicit mean-square error bounds and an order-optimal \emph{a priori} and \emph{a posteriori} parameter rules under H\"older-type source conditions. The framework is illustrated by partial observations and by the backward fractional heat equation. In the latter case, $$ B^*B=A_s^{-1}e^{-2tA_s}, $$ which permits a mode-wise description of the source condition, the singular-value decay, and the effective reconstruction bandwidth. We also study the local limit $s\to1^-$: after Bourgain--Brezis--Mironescu normalization, the fractional functionals $\Gamma$-converge in $L^2(\Omega)$ to the classical $H_0^1$-Tikhonov functional, and the corresponding minimizers converge strongly in $L^2(\Omega)$. Numerical experiments for the backward fractional heat problem illustrate the reconstruction procedure and the influence of the penalty order, and confirm the predicted mean-square convergence rate to within a few percent via Monte Carlo simulation, with Morozov's discrepancy principle attaining the same order-optimal rate a posteriori.
This work introduces a proximal stochastic subgradient method for minimizing the sum of an expected cost, whose integrand is potentially nonsmooth and nonconvex, and a lower semicontinuous, prox-bounded function. We target a broad class of integrands obeying a nonsmooth, localized variant of the descent lemma in the decision variable, a structural assumption that simultaneously covers smooth losses with Lipschitz gradient and differences of such losses with convex functions. At each iteration the expected cost is replaced by a sample average that is progressively refined, and the proximal-subgradient stepsize is selected by an Armijo-type line search enforcing a sufficient-decrease property up to stochastic errors induced by the sample-based approximation. This framework accommodates substantially more general problem formulations than existing methods, in particular, it requires neither (weak) convexity of the regularizer nor a uniform bound on the variance of the stochastic oracle, and our analysis yields convergence guarantees that are new even in the smooth setting. Specifically, we establish almost sure convergence of the sequence of function values and stationarity of every accumulation point of the trajectories under the relaxed requirement that the sample-size sequence be merely nondecreasing and unbounded, with no prescribed growth rate. Leveraging the Kurdyka-Lojasiewicz (KL) property, we further upgrade this subsequential guarantee to convergence of the whole trajectory to a single stationary point. Finally, for exponential-type KL desingularizing functions and polynomially growing sample sizes, we derive explicit polynomial convergence rates, up to a logarithmic factor, for both the function values and the iterates.
Felipe Atenas, Alejandro Jofré, Pedro Pérez-Aros et al.· 0 citations
This work develops a regularization analysis of non-accelerated and accelerated primal-dual methods for solving linear inverse problems in the presence of noisy data. We investigate a Condat-V\~u algorithm and an accelerated primal-dual hybrid gradient method in Hilbert spaces, with focus on quantifying the effect of data perturbations on the reconstruction error. For the non-accelerated scheme, we derive error estimates in terms of Bregman distances, whereas for the accelerated scheme we establish error estimates in norm. The study accommodates a general class of convex data fidelities satisfying suitable perturbation conditions, which are verified explicitly for equality constrained and Morozov regularization. For the non-accelerated method, the analysis is further extended to Banach spaces, taking into account non-Euclidean geometries and including a particular non-reflexive setting tailored to nonnegative solution reconstruction. The results recover known behavior in classical settings while extending the regularization analysis to these more general frameworks. Numerical experiments with representative regularizers, including sparsity and entropic models, support the theoretical findings and illustrate practical performance under noise.
Diana-Elena Mirciu, Martin Benning, Elena Resmerita· 0 citations
We study the problem of sampling from target distributions whose potentials are simultaneously non-smooth, subject to superlinear gradient growth, and non-convex. We introduce the Subgradient Tamed Unadjusted Langevin Algorithm (SG-TULA), a discretisation of the Langevin diffusion that operates directly on subgradients, without relying on computationally demanding smoothing procedures. To handle the superlinear regime, taming techniques are employed to produce a stable, explicit scheme. We derive non-asymptotic convergence bounds in Wasserstein-2 distance, with all constants tracked explicitly in terms of dimension and inverse temperature, improving upon the currently known rates for subgradient-based Langevin algorithms. We further provide excess risk estimates for the associated optimisation problem. We verify the assumptions, with explicit constants, for the regularized pretraining potential of a LLM in the GPT-2 lineage and the boosted coordinate-wise variant of SG-TULA pretrains the former competitively against finetuned AdamW and Muon, for which no comparable non-asymptotic guarantees are presently available.
Iosif Lytras, Nikolaos Makras, S. Sabanis· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
