This work introduces Gradient Control Rewards (GCR), an interpretable, control-inspired reward-design methodology for accelerating agent training by modulating the reward signal based on the temporal dynamics of system error, Inspired by classical control theory.
Abstract
Designing an optimal reward function is fundamental to achieving stability and efficiency in reinforcement learning (RL). This is particularly critical in robotics, where sparse rewards often provide insufficient guidance, necessitating the inclusion of auxiliary state information to facilitate meaningful exploration. This work introduces Gradient Control Rewards (GCR), an interpretable, control-inspired reward-design methodology for accelerating agent training by modulating the reward signal based on the temporal dynamics of system error. Inspired by classical control theory, GCR partitions the reward into three distinct components, state alignment, bias correction, and dynamic stability. These elements synergistically discourage the accumulation of error and excessive velocity toward objectives, facilitating the acquisition of a well-regulated action policy. GCR was evaluated across diverse environments, ranging from simple pendulum simulations to high-fidelity robotic scenarios and external physical validation. Experimental results demonstrate that GCR achieves competitive performance compared to both conventional reward functions and adaptive methods such as Bootstrapped Reward Shaping (BSRS). While alternative approaches exhibit performance degradation in stochastic, real-world-representative simulations, GCR maintains robustness and has been successfully validated in external physical environments. These findings suggest that GCR offers a practical and interpretable framework for deploying RL in control-oriented physical systems.
A unified analytical framework for comparing dynamic reward shaping and neighbouring adaptive reward mechanisms is introduced, which distinguishes parametric revision from state-dependent variation, separates additive shaping from reward replacement and reward-adjacent guidance, and organises existing methods along temporal, informational, and theoretical dimensions.
Reinforcement learning (RL) is a method of training artificial intelligence agents to make decisions
through trial and error, rewarding good behavior and penalizing bad behavior until the agent learns an
effective strategy. This study compares three widely used RL algorithms for continuous robotic control:
Proximal Policy Optimization (PPO), which learns by making small, cautious updates to avoid unstable
training; Soft Actor-Critic (SAC), which encourages broad exploration by rewarding the agent for trying
diverse strategies alongside completing the task; and Advantage Actor-Critic (A2C), which uses two
neural networks simultaneously - one to decide actions and one to evaluate them. Three experimental
conditions were evaluated across three robotic simulation environments of increasing difficulty: standard
task performance, energy-aware performance (where agents were equally penalized for excessive energy
use alongside task completion), and transfer learning (where agents pre-trained on standard rewards were
fine-tuned on energy-aware rewards). SAC consistently outperformed PPO and A2C on dense-reward
environments under both standard and energy-constrained conditions. Effect sizes indicated some nonsignificant
differences were nonetheless practically large. Most strikingly, A2C exhibited highly unstable
performance under energy-aware rewards on MountainCarContinuous-v0, achieving a mean reward of
-18,231 compared to near-zero values for PPO and SAC, suggesting that A2C may require additional
reward shaping or tuning before use in energy-constrained robotic applications. Transfer learning
experiments revealed that pre-training on standard rewards generally hurt rather than helped adaptation
to energy-aware objectives. These findings offer practical guidelines for algorithm selection in energyconstrained
robotic systems such as prosthetic hands and autonomous vehicles.
Om Herur· American Journal of Student...· 0 citations
This work proposes LaRes, a novel hybrid framework that achieves efficient policy learning through reward function search by leveraging large language models to generate the reward function population, guiding RL in policy learning.
Pengyi Li, Hongyao Tang, Jinbin Qiao et al.· Neural Information Processin...· 4 citations
This paper presents a systematic reward engineering methodology for training a Proximal Policy Optimization (PPO) quadrotor navigation policy in the Webots simulator, using a hierarchical architecture in which a PID controller handles low-level stabilization and a PPO policy issues velocity commands. We document the complete evolution of a composite ten-term reward function across seven versions (v5 through v11) and retrain the key versions with multiple independent training seeds. The multi-seed study revises the single-seed history: penalty-dominated configurations (v8, v10) fail across all seeds, while the strongest historical version proves seed-sensitive (v11: 32.2 +/− 15.8%). An ablation removing the continuous distance-shaping term from v11 yields 0% success across seven seeds, identifying that term as necessary for convergence. We further isolate a previously hidden co-factor: with the library-default exploration-noise initialization (sigma_0 = 1.0), sampled actions saturate the bounded action space, the exploration variance receives no learning gradient, and curriculum progression deadlocks regardless of reward design; initializing sigma_0 = 0.37 restores gradient flow. With this correction and a deterministic evaluation-gated curriculum, the final configuration is evaluated across the full curriculum rather than at a single operating point: across five independent training seeds under a deterministic protocol, it attains 95.0% ± 6.2% navigation success at Stage 0 conditions (2 m targets, no obstacles), 89.6% ± 6.9% at Stage 1 conditions (4 m, one obstacle), and 48.4% ± 10.8% at Stage 2 conditions (7 m, three obstacles). Reporting this difficulty curve, rather than a single headline value, exposes a substantial generalization gap whose dominant failure mode is obstacle collision (45–52% of episodes at Stage 2). Matched retraining of Soft Actor-Critic and TD3 baselines under identical reward and curriculum conditions yields one completed seed each both baselines show non-monotonic difficulty curves, and at Stage 2 conditions, TD3 (64.0%) exceeds PPO (48.4% ± 10.8%) while SAC (43.0%) falls just below it, whereas at Stage 0, PPO (95.0%) leads both, so the ranking is operating point-dependent on the current single-seed evidence. We conclude that a continuous shaping gradient and the exploration-noise initialization, interacting with the curriculum advancement criterion, determine convergence in continuous control deep reinforcement learning, and that reward polarity by itself does not.
A. Alkhodre, Mouhamad Alim Al-Amine, Yazed Alsaawy· Drones· 0 citations
MeRLa (Meta-Learned Reward Shaping), a principled framework that meta-learns a task-aware shaping function across auxiliary tasks before RLHF training, is introduced, providing theoretical guarantees for policy invariance, analyze representation drift sensitivity, and formally address incentive misalignment from entropy maximization.
A unified view of ARD in RL is provided by introducing a taxonomy, organized by the primary driver of the reward variation, that distinguishes external-feedback-driven reward updates from reward adaptations driven by endogenous within-run signals and those conditioned on exogenous context signals.
Raphaela Baybas, Carlo D'Eramo, Philipp Brune· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
