Noise scheduling and out-of-distribution generalisation in deep reinforcement learning for 6-DOF robotic grasping

Abstract

Abstract Exploration strategy is a fundamental but underexplored design choice in deep reinforcement learning for robotic manipulation. This paper presents a controlled empirical comparison of three off-policy configurations on a 6-DOF robotic grasping task using a Universal Robots UR5e in the RoboSuite simulation environment: Soft Actor-Critic (SAC) with automatic entropy tuning, Twin Delayed Deep Deterministic Policy Gradient (TD3) with piecewise linear noise decay, and TD3 with constant Gaussian noise. All configurations share identical network architectures, reward functions, and training budgets. The two TD3 variants differ only in the action-noise schedule and form the controlled comparison of the study, while SAC serves as a widely used entropy-regularised reference. Each configuration is trained over 10,000 episodes across five independent runs and evaluated over 1,000 deterministic episodes both within and beyond the training distribution. SAC achieves the highest in-distribution success rate (89.6%) and training efficiency, with a large effect size over both TD3 variants ( $$d = 1.32$$ and $$d = 1.47$$ ) and complete rank separation in area under the learning curve. Within the training distribution, the two TD3 variants are statistically indistinguishable ( $$d = 0.04$$ ). Under out-of-distribution evaluation, however, this contrast changes sharply: the noise-decay variant reaches a mean success rate of 43.1% against 32.3% for constant noise — an effect size of $$d = 1.08$$ , where the same comparison within the training distribution is negligible — together with the lowest cross-run variance of the three configurations and a more uniform distribution of success across the workspace. No pairwise difference reaches significance at five runs per configuration, and the comparisons are interpreted through effect sizes and confidence intervals. These results indicate that noise schedule design has a measurably different effect on in-distribution performance and spatial out-of-distribution generalisation, and that standard single-metric evaluation protocols may fail to capture meaningful differences between exploration strategies.