Robust disturbance-aware actor–critic reinforcement learning for multi-DOF robotic manipulators

Abstract

Building on recent insights that augmenting reinforcement‐learning policies with disturbance estimates improves robustness and sim-to-real transfer, this paper proposes a disturbance-aware actor–critic RL framework for high‐precision robotic manipulators. We derive the dynamics of manipulators ranging from two to six degrees of freedom and design a nonlinear disturbance observer that provides real-time estimates of lumped uncertainties. Unlike conventional feedforward-only designs, the proposed DACRL framework deeply integrates DOB information into the actor-critic’s learning state, cost function, and update laws to explicitly compensate for unknown dynamics and disturbances. A Lyapunov-based analysis proves that tracking errors, observer errors and neural-network weight errors remain uniformly ultimately bounded. Extensive simulations show that the proposed controller achieves sub-degree tracking errors across multiple DOF cases, with the 2-DOF example achieving steady-state errors below ±0.02 rad (Joint 1) and ±0.05 rad (Joint 2) and delivering faster convergence and smoother torques than conventional RL and DOB baselines. The disturbance-aware controller generalizes across trajectories and payloads, offering improved robustness while retaining learning flexibility. While the current study focuses on performance and robustness, future work could explore the integration of disturbance-observer-based control barrier functions to formally address safety constraints during the learning process. Simulation results suggest that the disturbance-aware controller can improve robustness while retaining learning flexibility. Real-platform validation remains an important direction for future work.