Hybrid Computed Torque Control and Soft Actor-Critic Framework for Sample-Efficient Robotic Manipulator Control

Manipulator Control Using DRL: Sim-to-Real Validation on a 3-DoF Arm

Deep reinforcement learning is emerging as a powerful alternative to traditional inverse kinematics for controlling robotic manipulators. By learning optimal actions through interaction with the environment, it enables adaptable and precise control in complex continuous workspaces, making it suitable for dynamic manipulator operations. This paper proposes an actor–critic deep reinforcement learning framework for manipulator control in a continuous workspace. Conventional inverse kinematics solutions can become computationally complex for high-degree-of-freedom manipulators or complex kinematic structures. Accurately generating joint angles by deriving configuration-specific equations therefore remains a persistent challenge. This study employs deep deterministic policy gradient and twin delayed deep deterministic policy gradient algorithms to directly predict joint angles for target-reaching tasks, eliminating reliance on conventional inverse kinematics formulations. A key contribution is a simple yet effective linear reward function that supports stable convergence in continuous action spaces. The proposed framework is implemented using ROS2 and Gazebo simulation and validated on a custom-built physical manipulator. The work addresses complex equation-based control through reinforcement learning and demonstrates proof of concept on a 3-degree-of-freedom manipulator. Experimental results show high task success rates of 95.77% for the deep deterministic policy gradient algorithm and 94.71% for the twin delayed deep deterministic policy gradient algorithm, with accurate end-effector positioning within 0.01 m. These results confirm the effectiveness of the proposed framework for accurate and reliable manipulator control with reduced model complexity and improved real-world applicability. Received: 13 September 2025 | Revised: 14 April 2026 | Accepted: 10 June 2026 Conflicts of Interest The authors declare that they have no conflicts of interest to this work. Data Availability Statement Data are available from the corresponding author upon reasonable request. Author Contribution Statement Shivkumar Sankaralingam: Conceptualization, Methodology, Software, Validation, Formal analysis, Investigation, Writing – original draft, Writing – review & editing, Visualization. NippunKumaar Arulmani Angamuthu: Conceptualization, Methodology, Formal analysis, Investigation, Writing – review & editing, Visualization, Supervision. Suja Palaniswamy: Writing – review & editing, Visualization, Project administration, Funding acquisition.

Reinforcement Learning Based Optimal Control: A Survey of Adaptive Dynamic Programming for Manipulators and Wheeled Mobile Robots

Residual Deep Reinforcement Learning-Based Computed Torque Control for a Cable-Driven Lower-Limb Rehabilitation Robot under Disturbances and Parametric Uncertainties

Bounded residual learning is supported as a practical robustness-enhancement strategy for simulation-based rehabilitation robot control and will motivate further constraint-aware and experimental validation.

Reinforcement Learning for Real-Time Auto-Tuning of Robot’s Controllers

This research introduces an innovative control technique for Series Elastic Actuators (SEAs) that utilizes Reinforcement Learning (RL) to address the shortcomings of previously fixed-gain adaptive controllers, which are a hybrid of State Feedback Control (SFC) and Model Reference Adaptive Control (MRAC) by using Lyapunov Stability Analysis. This controller is optimized by adjusting the adaptation factor. b. This study presents an intelligent agent based on reinforcement learning to find the value of b with a dynamic auto-tuner. It trains via the Soft Actor-Critic (SAC) algorithm for 100,000 time steps. A comparison between the two methods was presented according to simulation results under different conditions; the RL-based controller shows much better tracking accuracy, how quickly it reaches the target output, and how little control torque it uses, where the agent's policy could automatically adjust in real-time based on system conditions, such as uncertainties and disturbances, where it has a settling time of 1.7 seconds, while the fixed parameter controller has a 1.95-second settling time, resulting in a reduction of 15.3%. It also lowers the control torque caused by disturbances by 19.5% compared to the fixed parameter controller, which has a control torque of 3.99 Nm, while the maximum control torque for the RL-optimized controller is 3.21 Nm.

Neural-Augmented Torque Control for Robotic Manipulators: Modeling, Learning, and Real-Time Performance

This thesis investigates advanced modeling and control strategies for robotic manipulators, focusing on the DLR-HIT II robotic hand and the KUKA LBR iiwa. It presents three core contributions that integrate simulation, model-based control, and data-driven methods to improve torque and position control under uncertainties and disturbances. First, a dual-platform simulation framework is developed using MATLAB Simscape Multibody and CoppeliaSim. The system accurately models the DLR-HIT II hand’s kinematics and dynamics, enabling both control validation and realistic interaction with virtual environments. The use of Unified Robot Description Format (URDF)-based modeling sup-ports reusability and modular analysis. Second, a physics-informed neural network (PINN) is proposed for direct torque and position control. This method uses only time and joint position inputs, internally computes derivatives, and generalizes well across various trajectory types. It eliminates the need for separate feedback controllers and shows strong robustness under disturbances, while maintaining low computational cost. Third, a Spike-Aware Hybrid Torque Control (SA-HTC) architecture is enhanced with a feedforward neural network (NN) trained offline. The network learns to improve Computed Torque Control (CTC) for unmodeled effects, friction, and external forces by refining the CTC torque output in real time. Simulation results across diverse trajectories and noise levels demonstrate that the SA-HTC method significantly improves tracking accuracy and robustness compared to classical CTC. To support deployment on position-controlled hardware, a physics-informed torque-to-position interface is introduced and compared with virtual stiffness and admittance wrappers, yielding lower steady-state bias, reduced phase lag, lower tracking error, and robust cross-trajectory generalization. Thus, these contributions advance the integration of learning-based and model-based control strategies in robotics. The results highlight scalable and efficient methods for accurate trajectory tracking and torque control, offering practical potential for robotic manipulation and automation applications.

Hybrid Reinforcement Learning for Efficient Training of a Mobile Robot

Sample effective and stable training remains a key challenge in reinforcement learning (RL), especially for real-world applications such as mobile robot control where data collection is time-consuming and failures may be hazardous.Building on the residual reinforcement learning paradigm, this work presents, to the best of our knowledge, one of the first detailed physical studies of a residual Soft Actor-Critic (SAC) controller for camera-based lane following on a mobile robot. We combine an established stable, but sub-optimal lateral P-controller with a regularized SAC agent in a hybrid architecture. The classical controller provides baseline stability and rapid initial learning, while the RL agent learns residual corrections to improve performance. We employ a PID-inspired reward function and quadratic policy output regularization to ensure smooth control actions and effective sim-to-real transfer.The hybrid controller design enables rapid training convergence, requiring only a few epochs and outperforming the pure RL approach by two orders of magnitude in sample efficiency. This enables efficient hyperparameter tuning in simulation and opens the door to future learning directly on physical robots. Fine-tuning with only a few dozen real-world laps achieved robust transfer to the physical robot, maintaining the same architecture and hyperparameters. The method generalized effectively to new scenarios, such as lane changes.