#artificial intelligence #machine learning Preprint

On the Convergence of Semi-Gradient TD: A Mean Field Perspective

Haruki Settai Tadashi Kozuno Shinji Ito

Oct 2026 · 0 citations · 15 references

Computer Science

TL;DR

Under this condition, it is proved exponential convergence to a unique fixed point and its bias relative to the unregularized Bellman solution and it is shown that the same mechanism extends to entropy-regularized off-policy control through soft Q-learning.

Abstract

Semi-gradient temporal-difference learning is a core algorithmic component of reinforcement learning, but bootstrapping with function approximation generally induces a non-conservative vector field and can lead to instability. We analyze this effect for two-layer neural networks in the mean-field regime and identify a simple stability mechanism in which entropy-induced relaxation dominates the sensitivity of the moving bootstrap target. Under this condition, we prove exponential convergence to a unique fixed point and quantify its bias relative to the unregularized Bellman solution. We also show that the same mechanism extends to entropy-regularized off-policy control through soft Q-learning.

DOI arXiv Semantic Scholar

View source

Similar papers

#machine learning Preprint Sep 2026

Exact information accounting for SGD methods

An exact, information-theoretic analysis of stochastic gradient descent and its variants is given, showing that a preconditioned SGD step is the posterior-mean update of a Gaussian Bayes model, and that its one-step regret splits into an intrinsic-time cost and a change in comparator information.

Akshay Balsubramani · 0 citations

Finite-Time Analysis of Minimax Q-Learning

Narim Jeong, Donghwan Lee · 0 citations

#machine learning Preprint Oct 2026

Mind the Drift: Diagonal Linear Networks Under Large Learning Rates

Large learning rates can qualitatively change the trajectory of neural network training, often pushing optimization into regimes far from classical gradient-flow behavior. The Edge of Stability (EoS) offers a valuable lens on the dynamics such learning rates induce. We study corresponding dynamics in diagonal linear ne...

Aniket Sanyal, Tom Jacobs, Rebekka Burkholz · 0 citations

HANQ: Hypergradients, Asymmetry, and Normalization for Fast and Stable Deep Q -Learning

Braham Snyder, Chen-Yu Wei · 0 citations

Viscosity Convergence Analysis for Deep Q-Networks

This work studies a spatially-coupled monotone ResNet architecture whose one-step operator is a non-negative local aggregation, designed to satisfy monotonicity and stability at the operator level, and obtains operator-level convergence to the viscosity solution via the Barles–Souganidis framework.

Qian Qi · 1 citation

#machine learning Preprint Oct 2026

Entropy-Regularized Mean-Field Games with Average-Reward Criterion: Span-Seminorm Contractions and Model-Free Learning

We study stationary discrete-time mean-field games with finite state and action spaces under the average-reward criterion, with an entropy-regularized objective and population-dependent dynamics and rewards. Under a minorization condition, the soft Q-operator is a span-seminorm contraction for the full transition kerne...

Gunes Akbas, Naci Saldi, Berkay Anahtarci et al. · 0 citations