To address the discontinuous ℓ0-type sparsity penalty, smooth non-convex sparsity approximations are introduced that preserve differentiability while retaining sparse threshold behavior and characterize a quasi-threshold sparse structure of the resulting optimal feedback policies in non-affine stochastic systems with control-dependent noise.
Abstract
Sparse ergodic control provides a natural framework for long-run stochastic decision-making under resource constraints. Existing formulations, however, are typically restricted to control-affine systems with control-independent diffusion. When the diffusion coefficient depends explicitly on the control input, the associated ergodic Hamilton–Jacobi–Bellman (HJB) equation becomes non-separable through the term trax, u∇2V, so classical arguments based on control-affine separability no longer apply directly. In this work, we study sparse ergodic control of stochastic systems with control-dependent diffusion and nonlinear dynamics within a viscosity-solution and learning-based framework. To address the discontinuous ℓ0-type sparsity penalty, we introduce smooth non-convex sparsity approximations that preserve differentiability while retaining sparse threshold behavior. Within a viscosity-solution framework, we analyze the existence and uniqueness properties of the associated ergodic pair and establish localized approximation error estimates for the smooth approximation. We further characterize a quasi-threshold sparse structure of the resulting optimal feedback policies in non-affine stochastic systems with control-dependent noise. On the computational side, we develop a Physics-Informed Neural Network (PINN)-based solver with adaptive residual-driven sampling for high-dimensional sparse ergodic HJB equations, together with a distributed monotone-inspired iterative scheme for weakly coupled multi-agent systems. Numerical experiments on multi-robot swarm navigation and renewable-integrated smart-grid control demonstrate that the proposed methods produce sparse control policies while preserving stable long-run performance under stochastic disturbances.
We study continuous-time and possibly high-dimensional stochastic control problems where drift coefficients and running reward functions are unknown. Due to these missing model primitives, we take the exploratory, reinforcement learning (RL) framework of Wang, Zariphopoulou, and Zhou(2020) with relaxed controls and entropy regularization. The objective is to develop theoretically grounded, efficient and scalable RL algorithms to learn both the optimal value functions (which also solve the exploratory HJB equation) and optimal exploratory feedback control policies. When the diffusion coefficients do not contain control, we employ probabilistic representations of both the optimal value function and its gradient based on an auxiliary state process depending only on the diffusion part of the original dynamics. With a delicate analysis on some properly defined mappings and their fixed points, this leads to the introduction of our policy iteration algorithms and their convergence. We demonstrate the performance of our algorithms through various numerical examples. Finally, we study a special control-dependent diffusion case where probability representation of the Hessian is called for.
Jing-Sheng Ma, Gao-Zhan Wang, Jian-Feng Zhang et al.· 0 citations
We estimate convergence rates of value functions for particle approximations of a controlled nonlinear filtering problem. The state is a McKean--Vlasov diffusion on the flat torus, driven by hidden idiosyncratic noise and observed common noise. The filter---the conditional law of the state given the observations---serves as the state variable of the control problem, and the associated value function solves a second-order Hamilton--Jacobi--Bellman equation on the Wasserstein space. We approximate this problem by a centralized \(N\)-particle control problem with independent idiosyncratic noises and a common observation noise. The framework accommodates nonseparable rewards and controlled drifts. Since a single control is applied to the entire population, the Hamiltonian is defined by an optimization performed after integration over the population. Under smoothness of the data, uniform ellipticity, and regularity of this Hamiltonian, we establish uniform value-function error bounds of order \(N^{-1/6}\) for \(d=1\), \(N^{-1/6}(\log N)^{1/3}\) for \(d=2\), and \(N^{-1/(3d)}\) for \(d>2\). The proof combines a translation lift in the common-noise direction, Fourier--Wasserstein inf- and sup-convolutions, viscosity comparison, and particle derivative estimates uniform in \(N\).
Erhan Bayraktar, Ibrahim Ekren, Xi-Hao He et al.· 0 citations
A regularized PINNs framework that incorporates Tikhonov regularization to solve terminal-state tracking optimal control constrained by parabolic partial differential equations is proposed, establishing a consistency result showing that PINNs minimizers nearly attain the continuous regularized objective under residual and quadrature approximation assumptions.
We study objective-only optimization of possibly nonconvex and nonsmooth functionals over probability measures on a separable Hilbert space, allowing the optimizer to be intrinsically non-Dirac. We introduce SCMO (Stochastic Control Measure Optimizer), a gradient-free particle method derived from entropy regularized stochastic control. After finite-particle and Galerkin approximations, a Cole--Hopf transform represents the optimal feedback as a Gibbs-weighted terminal displacement. SCMO approximates this feedback by sampling context clouds, replacing one particle at a time with candidate draws, scoring the resulting empirical measures, and applying exponential reweighting. SCMO uses separate covariances for candidate proposals and particle updates, termed matched when equal and nonmatched otherwise; our analysis covers both settings. In the matched case, we establish PDE-free qualitative convergence and a projection-first quantitative bound separating particle, Galerkin-projection, and entropy-regularization errors. For the practical multi-context implementation with nonmatched covariance, we prove finite-candidate consistency and show that the signed, curvature-dependent effect of nonmatched covariance can reduce the resulting upper bound on the approximation error. Finally, experiments on function-space, trajectory-law, and contact-rich manipulation problems show that SCMO handles nonsmooth and nonconvex objectives, escapes suboptimal local basins, and recovers prescribed multimodal, non-Dirac law structure. Code for reproducing our experimental results is available at https://github.com/HenryCHEUNG7373/SCMO.
Stochastic quantization defines a Euclidean quantum field theory as the equilibrium of a fictitious-time Langevin dynamics, which reaches the Gibbs measure asymptotically and is formulated as a finite-time stochastic optimal control problem.
We propose a novel Stochastic Nonlinear Model Predictive Control (SNMPC) framework for nonlinear systems with additive noise. Building on recent advances in nonlinear uncertainty propagation, we show that the state distribution of the system can be tractably approximated over time by Gaussian mixture distributions, with formal error bounds in Wasserstein distance. This representation yields closed-form expressions for expected costs and chance constraints, which become exact for affine constraints and exact up to a constant for quadratic costs. Consequently, the resulting control problem can be solved efficiently via nonlinear programming, while providing formal open-loop guarantees of correctness and asymptotic optimality. Experiments on a set of benchmarks demonstrate that the proposed approach compares favorably with existing methods in nonlinear settings with multi-modal disturbances, where standard approaches lead to poorly scaled solutions and unsafe or overly conservative control actions.
Konstantinos Prattis, Luca Laurenti, A. Dabiri· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
