#artificial intelligence #machine learning Preprint

On BatchNorm Forward Modes in Value-Based Reinforcement Learning

Daniel Palenicek Mikael Henaff Scott Fujimoto Koustuv Sinha

Sep 2026 · 0 citations · 23 references

Computer Science

TL;DR

The results show that carefully configured BN can substantially improve discrete-action value learning, and that its forward protocols are an essential part of the algorithm specification.

Abstract

Batch normalization (BN) substantially improves sample efficiency in continuous-control actor-critic methods such as CrossQ, yet recent studies report performance degradation in discrete-action value learning on Atari. These failures are surprising because discrete Q-networks lack the action-input distribution mismatch identified by CrossQ. We show for target-based C51 and target-free PQN that the simple choice between running and batch statistics at specific forward passes can reverse this degradation. In C51, switching the BN bootstrap forward to batch-statistic mode significantly improves performance over unnormalized and LayerNorm baselines and scales stably with update-to-data ratios up to 12. In PQN, using batch-statistics for both action selection and bootstrapping recovers performance from the failing running-statistic configuration. Across 26 Atari games at 400M frames, this configuration achieves a higher final aggregate score than PQN with LayerNorm. Our results show that carefully configured BN can substantially improve discrete-action value learning, and that its forward protocols are an essential part of the algorithm specification.

DOI arXiv Semantic Scholar

View source

Similar papers

#machine learning Preprint Sep 2026

From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning

Q-Target Pretrained Transformers is proposed, which keeps the context-conditioned Transformer architecture but replaces behavior cloning with a Bellman-style Q-target objective, and learns to use rewards and transitions in the context to estimate action values, rather than simply imitating the behavior policy.

Yi-Chen Lin, Xu-Yuan Xiong, Xue Wang et al. · 0 citations

Preprint Aug 2026

Best Practice Critic Optimization

Best Practice Critic Optimization (BPCO) is developed, a recipe that combines DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation and shows that a carefully designed critic provides a reliable alternative to...

Penghui Qi, Xiang-Xin Zhou, Wee-Sun Lee · 3 citations · ⚡1

#machine learning Preprint Sep 2026

On-Policy or Off-Policy Learning? A Systematic Study of Distillation Dynamics

The results challenge the view that on-policy rollouts are inherently preferable and show that their value depends critically on the objective, evaluation setting, and optimisation hyperparameters.

Julianna Piskorz, Antonin Berthon, M. van der Schaar · 1 citation

#machine learning Preprint Oct 2026

Why On-Policy Distillation Sometimes Fails: Vanishing Learning Signals

A local recovery guarantee for teachers sufficiently close to the initial student in a shared parameterization under regularity conditions is proved, offering a conditional explanation for the success of self-RL teachers in the authors' experiments.

Lei Zhao, Qi-Chao Zhao, Bo-Wen Zuo et al. · 0 citations

#machine learning Preprint Sep 2026

Statistical Convergence of Transformer Encoder-Accelerated Robust Reinforcement Learning

Obtaining the optimal action-value function in Markov decision processes is computationally intensive in large state--action spaces. In this study, we present statistically rigorous convergence results for a robust reinforcement learning algorithm warm-started by a transformer-based action-value function prediction, wh...

Suman Banerjee, Hiroyasu Tsukamoto · 0 citations

#artificial intelligence Preprint Oct 2026

Follow the Winners: Conservative Policy Improvement with the Cross-Entropy Method for Critic-Free RFT

Critic-free reinforcement fine-tuning (RFT) for agentic large language models is often done through GRPO-style methods, which compute a group baseline over repeated rollouts to reduce target variance. However, this setup is ill-suited to agents acting in stateful environments such as live services or security sandboxes...

Joery A. de Vries, Neil D. Lawrence, Zhen-Wen Dai · 0 citations