Skip to content

Sufficiency of Zeroth-Order Reward Shaping for Policy Gradient in Stabilization Control

Sep 2026 · 0 citations · 52 references
Computer Science

TL;DR

This work theoretically and empirically demonstrate that policy gradient methods can successfully solve stabilization tasks without first-order reward terms, and confirms that reward functions must be zeroth-order complete over goal-relevant coordinates, while the first-order state remains necessary in the policy observation under low-dissipation assumptions.

Abstract

Reward shaping is fundamental to modern robotic control with deep reinforcement learning (RL), yet practitioners still rely heavily on heuristic principles borrowed from classical optimal control and trajectory optimization. Existing methods rarely distinguish reward terms that are intrinsic to the control objective from numerical regularizers, leading to brittle hyperparameter tuning. To determine which quantities a reward must contain, we study the stabilization control problem with a focus on zeroth-order (configuration) and first-order (velocity) information. We theoretically and empirically demonstrate that policy gradient methods can successfully solve stabilization tasks without first-order reward terms, adding such terms can instead introduce severe sensitivity as their scale grows. Conversely, our findings confirm that reward functions must be zeroth-order complete over goal-relevant coordinates, while the first-order state remains necessary in the policy observation under our low-dissipation assumptions. Overall, these results provide actionable and principled guidance for reward design in robotic RL.

View source

Similar papers

Preprint Sep 2026

SafePG: Safe and Globally Optimal Reinforcement Learning with Hard Constraints

An optimal and convergent model-free policy gradient (PG) reinforcement learning (RL) reinforcement learning framework for controlling nonlinear dynamical systems under hard safety constraints is presented and a model-free PG algorithm based on stochastic gradient ascent is developed.

Vipul K. Sharma, Wesley A. Suttle, S. Sivaranjani · 0 citations
#artificial intelligence Preprint Oct 2026

Higher-Order Action Supervision Makes A Strong Policy Class

Modern data-driven decision-making methods, such as imitation learning (IL) and reinforcement learning (RL), have achieved great success in solving many complex tasks. However, these methods often suffer from serious control instability and robustness issues when applied in real-world applications such as robotics and...

Peng Cheng, Yun-Xian Hou, Zhi Zhou et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Loop-Free Inverse Reinforcement Learning via Sequential Value Recovery with Q-Score Matching

Inverse Reinforcement Learning (IRL) aims to recover a reward function that explains expert demonstrations. Existing IRL methods typically rely on a bi-level optimization procedure that alternates between reward learning and policy optimization, leading to substantial computational burden and training instability. In t...

Yang Chen, Yi-Tan Zhang, Michael J. Witbrock et al. · 0 citations
Open access Aug 2026

Shaping Gradient and Exploration-Noise Initialization, Not Reward Polarity, Determine Convergence in Deep Reinforcement Learning for Autonomous Quadrotor Navigation and Obstacle Avoidance

It is concluded that a continuous shaping gradient and the exploration-noise initialization, interacting with the curriculum advancement criterion, determine convergence in continuous control deep reinforcement learning, and that reward polarity by itself does not.

A. Alkhodre, Mouhamad Alim Al-Amine, Yazed Alsaawy · 0 citations
Sep 2026

GR2PO: Group Relative Return Policy Optimization for Continuous Robot Control

Group Relative Return Policy Optimization (GR2PO), a critic-free reinforcement learning framework for continuous robot control, is proposed and significantly outperforms critic-free baselines that use immediate rewards and performs competitively against state-of-the-art actor-critic methods.

Pengqin Wang, Qi-Ming Zhang, Shao-Jie Shen et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Optimal Control with Learned Critics under Unmodeled State Dependencies

A learning-based MPC framework that combines the data efficiency and structure of local model-based planning with learned components that compensate for incomplete dynamics and finite-horizon myopia and is integrated into a robotics simulator, enabling scalable model-based reinforcement learning under incomplete dynami...

P. Schöch, Markus Ryll · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.