A physics-aware, end-to-end deep reinforcement learning (DRL) approach that acts directly on low-level body inputs, total thrust and body torques, and closes the loop through a high-fidelity Simulink environment is investigated.
Abstract
Unmanned aerial vehicles (UAVs), particularly quadcopters, present unique challenges for autonomous control due to their underactuated dynamics: only four available control inputs must govern six degrees of freedom. This paper investigates a physics-aware, end-to-end deep reinforcement learning (DRL) approach that acts directly on low-level body inputs, total thrust and body torques $(T, \tau_x, \tau_y, \tau_z)$, and closes the loop through a high-fidelity Simulink environment. Our simulator integrates a 12-state rigid-body model (MATLAB Level-2 S-Function) with (i) an Action2RPM allocation based on the Moore-Penrose pseudo-inverse of a coefficient matrix derived from thrust and drag terms, and (ii) first-order actuator dynamics for each motor (time constant $T_m = 0.076$ s), including rotor gyroscopic coupling. A shaped reward balances goal-reaching and stability using an exponential position well, attitude penalties, and quadratic velocity costs. Four DRL algorithms, DDPG, TD3, PPO, and SAC, are evaluated in two stages: (S1) thrust-only hover and (S2) hover with pitch torque and a translated goal. Results show that SAC and TD3 achieve superior stability and exploration efficiency, while PPO is less sample-efficient. The study highlights the significance of modeling actuator lags and aerodynamic moments for stable low-level control and provides a reproducible benchmark for quadcopter DRL.
This paper explores and evaluates both traditional and learning-based methods for spacecraft control and coupled robotic arm manipulation in a microgravity environment. Space junk, debris, and out-of-control satellites currently floating in orbit pose a severe risk to critical space assets, necessitating debris removal systems. Controlling a robotic arm and base vehicle in zero-gravity environments poses a significant challenge due to the inherent physical coupling of the system’s dynamics. This paper is concerned with a 3-DOF planar spacecraft with a 2-DOF manipulator approaching and intercepting a rotating object in 2D space. It establishes a cascaded Proportional-Integral-Derivative (PID) controller as a baseline benchmark before focusing on comparing optimal and learning-based con-trol strategies, specifically a unified 5-Degree-of-Freedom non-linear Model Predictive Control (MPC) and Deep Reinforcement Learning (DRL). The analysis evaluates steady-state precision, dynamic stability, and approach trajectories to determine the operational suitability of each methodology for target interception and maintained contact.
Patrick Coulon, Charles East, Luke Busse et al.· National Aerospace and Elect...· 0 citations
Deep reinforcement learning for autonomous unmanned aerial vehicle control has largely been demonstrated with multirotor platforms and high-level machine-learning frameworks. This study presents a Dueling Double Deep Q-Network (D3QN) training pipeline implemented in Rust without an external machine-learning library and integrated with Godot 4 through GDExtension for fixed wing flight control. The controller addresses fixed-wing requirements, including airspeed maintenance, lift management, throttle regulation, stall avoidance, and coordinated turning. The network combines a duelling architecture, double Q-learning, prioritised experience replay, and three-step returns in a 512→256 hidden-layer configuration containing 142,088 parameters for the 16-dimensional input case. The agent selects among seven discrete actions and supports both 12-dimensional and 16-dimensional observation spaces through a cross-dimensional weight-transfer procedure. Training was conducted for 200 episodes using four random seeds. Across seeds, the mean best episodic reward was 4825±40, while the coefficient of variation for best reward was 0.8%. In the final 30 episodes, no crashes were recorded, although completion rates varied substantially between seeds.
Airspeed remained within ±8 m/s of the 50 m/s target. Batch-64 gradient updates required less than 1 ms, representing an approximately 35-fold reduction in latency relative to the preceding GDScript implementation, and the reported runtime memory footprint remained below 50 MB. These findings support the feasibility of native Rust-based D3QN training for real-time fixed-wing simulation, while the observed inter-seed variability indicates that reward shaping and convergence robustness require further evaluation.
Saugat Chaudhary Tharu, Shrutika Ojha, Rija Bhomi et al.· Journal of Advances in Mathe...· 0 citations
This paper presents an advanced deep reinforcement learning (DRL) framework for precise trajectory tracking control of an underactuated 2-degree-of-freedom (2-DOF) helicopter system using the twin delayed deep deterministic policy gradient (TD3) algorithm. The 2-DOF helicopter serves as a benchmark for nonlinear, coupled, and underactuated systems, posing significant challenges for conventional control approaches. Both classical linear and nonlinear control methods provide baseline solutions; however, their performance often degrades in the presence of parameter variations, uncertainties, and external disturbances. To overcome the severe value overestimation errors caused by aerodynamic cross-coupling in standard actor-critic architectures, a model-free TD3-based controller is developed, incorporating an artificial potential field-inspired reward function to simultaneously optimize tracking accuracy, energy efficiency, and control smoothness. Compared with standard DRL approaches such as the deep deterministic policy gradient (DDPG), the TD3 algorithm addresses key limitations by employing twin critics to reduce overestimation bias, delayed policy updates to improve training stability, and target policy smoothing to enhance robustness. Comprehensive simulations conducted in a MATLAB/Simulink environment demonstrate the superior performance of the proposed TD3 controller compared to classical and intelligent approaches, including proportional-integral-derivative (PID), fuzzy PD + I, and fuzzy PD + FF controllers. For multi-step trajectory tracking, TD3 reduces overshoot to 4.6% (pitch) and 3.8% (yaw) compared to 22.4% and 18.7% for PID, while decreasing settling time by up to 66%. The steady-state error is reduced to 0.18° (pitch) and 0.15° (yaw), representing improvements exceeding 80% over PID. In addition, TD3 minimizes cross-coupling effects by over 60%, enabling effective decoupled control of pitch and yaw dynamics. Under complex trajectories and disturbance conditions, including ± 10% parametric uncertainties and external torque disturbances, the TD3 controller consistently achieves the lowest tracking errors, fastest convergence, and smoothest control signals, reducing control variation by up to 65% compared to conventional methods. These results highlight the effectiveness of TD3 for controlling nonlinear and underactuated systems and provide a solid foundation for future experimental validation and real-world deployment in aerial robotic platforms operating in uncertain environments.
Zied Ben Hazem, Muhammed Özdemir, Firas Saidi et al.· Discover Robotics· 1 citation
Yaw regulation of biomimetic underwater robots is complicated by flexible body motion, nonlinear hydrodynamics, and coupled actuation. This study examines whether a policy trained in simulation can be deployed on an existing robotic sea lion (RSL) without changing its hardware or low-level controllers. A deep deterministic policy gradient (DDPG) controller was formulated from measurable states and available actuator commands and trained in Webots using a model calibrated from previous tank tests. Four manually selected reward-weight settings and command update rates of 1, 2, 5, and 10 Hz were examined as deployment-oriented sensitivity comparisons, after which a 5 Hz policy was evaluated in six tank trials. Performance was reanalyzed using the circular-angle mean absolute error (MAE) and root mean square error (RMSE). In the two straight-swimming trials, the per-trial MAE was 1.01–1.12°, and the RMSE was 1.10–1.46°. In the four turning trials, evaluated from the first target crossing to the end of each record, the MAE was 1.71–3.63°, the RMSE was 2.10–4.20°, and the maximum overshoot was 2.99–7.70°. Despite the transient differences between simulations and experiments, the controller regulated the robot toward the target headings in all six tank trials. These results demonstrate the successful sim-to-real deployment of reinforcement-learning-based yaw control under low-frequency communication constraints and provide experimental evidence for its application to biomimetic underwater robots.
Ze-Yi Zhang, Yu-Hong Liu, Shuang-Tao Liu et al.· Journal of Marine Science an...· 0 citations
This paper presents the development and implementation of an end-to-end control framework for a quadruped walking robot based on deep reinforcement learning. The primary objective of the study is to design and verify a control system capable of autonomously generating locomotion strategies. A model of the walking robot was developed using the Simscape Multibody toolbox, providing a physics-based simulation environment for training and evaluation. The proposed control approach employs a deep reinforcement learning agent trained using the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm. The agent learns locomotion behaviors directly from interactions with the simulated environment, without relying on predefined gait trajectories or manually designed control laws. Through iterative training, the agent optimizes its policy to maximize a predefined reward function, enabling the robot to discover efficient and stable movement patterns. Simulation results demonstrate that the TD3-based approach is highly effective for continuous control tasks involving systems with complex nonlinear dynamics. The trained agent successfully learned locomotion strategies, including dynamic gaits with flight phases, highlighting the ability of reinforcement learning methods to handle naturally unstable behaviors that are difficult to design using classical control techniques.
Filip Połatyński, Paweł Skruch· International Conference on...· 0 citations
Motion control of tilt-servo fully vectoring UAVs (TSFV-UAVs) is challenging due to their highly nonlinear dynamics. This paper proposes a reinforcement-learning-based training framework for the hierarchical control architecture of TSFV-UAVs. The framework introduces an adaptive guidance mechanism and a varying-gradient reward function to improve training convergence. In addition, a wrench residual penalty term is incorporated into the reward function to help the agent identify the boundary of the reachable wrench set, thereby reducing the frequency of actuator saturation and improving the task success rate of the UAV. The results show that the proposed method not only effectively improves training convergence, but also enables the trained agent controller to achieve significant improvements in tracking performance and task success rate (93.0%), while exhibiting good robustness. Finally, hardware-in-the-loop experiments verify the practical deployability of the trained controller on embedded systems.
