A fully computational framework focusing on the modeling and simulation of a spatio-temporal sensory system to autonomously generate the Kármán gait is proposed, providing a robust algorithmic blueprint for future physical deployments in complex aquatic environments.
Abstract
Navigating in unsteady wake flows, such as Kármán vortex streets, presents a formidable challenge for biomimetic autonomous underwater vehicles. Biological fish achieve this by utilizing their lateral line sensory systems to perceive local flow gradients and adopting an energy-efficient swimming pattern known as the Kármán gait. To translate this biological phenomenon into a practical robotics engineering solution, this paper proposes a fully computational framework focusing on the modeling and simulation of a spatio-temporal sensory system to autonomously generate the Kármán gait. To overcome the unrealistic assumption of full-state observability common in existing reinforcement learning studies, we model a multi-point lateral line array coupled with a frame-stacking mechanism. This allows the simulated agent to reconstruct the spatio-temporal topology of the surrounding unsteady flow relying exclusively on local pressure and velocity gradients. The sensory model is integrated with a spatio-temporal perceptual twin delayed deep deterministic policy gradient (STP-TD3) algorithm, which drives a Hopf-oscillator-based central pattern generator. Through rigorous high-fidelity computational fluid dynamics simulations, we quantitatively evaluate the autonomous emergence of the Kármán gait by assessing the agent’s kinematic energy proxy-mapped from joint actuation effort. Results reveal that the agent expends significantly less mechanical effort navigating through the turbulent vortex street compared to swimming in steady water, suggesting the active exploitation of the local wake dynamics. The results theoretically underscore the necessity of distributed STP for biomimetic robots, providing a robust algorithmic blueprint for future physical deployments in complex aquatic environments.
Autonomous locomotion in robotic fish requires task-dependent control capabilities under changing environmental conditions. This paper proposes a hierarchical simulation-based control framework for a two-joint robotic fish in a two-dimensional (2D) planar environment. This framework integrates the twin delayed deep deterministic policy gradient (TD3) algorithm with a sensory-feedback central pattern generator (CPG). A nonlinear planar dynamic model is designed as the learning environment, and a CPG network generates rhythmic undulatory swimming. The CPG network generates smooth locomotor patterns, while the TD3 policy performs high-level neuromotor modulation for task-dependent behavior. In the target-reaching benchmark, TD3–CPG achieves a 100.0% success rate with a Wilson 95% confidence interval (CI) of [96.30%, 100.00%], outperforming benchmark models. The proposed controller is also evaluated with obstacle avoidance in target reaching and station keeping under current disturbances. In circular obstacle avoidance, TD3–CPG achieves a 98.0% success rate and a 98.0% safe-pass rate, whereas the multiple rectangular obstacle scenarios yield an overall success rate of 91.7% over 96 trials. In station keeping, the controller achieves stay ratios of 87.57 ± 12.81% under constant current and 96.88 ± 10.79% under gust current, while keeping the mean target distance below the 0.25 m station keeping radius in both cases. Within the adopted 2D planar simulation environment, the obtained results demonstrate that the proposed method exhibits task-dependent maneuvering performance within the evaluated scenarios.
Gonca Ozmen Koca, D. Korkmaz, Cafer Bal et al.· Biomimetics· 0 citations
This paper demonstrates how deep reinforcement learning (DRL) enables adaptive locomotion of snake-like robots in dynamically changing viscous environments, overcoming the inherent performance limitations of classical predefined control methods. The lack of direct onboard sensors for fluid properties necessitates formulating this task as a partially observable Markov decision process. By employing an asymmetric actor-critic framework, a teacher policy trained using privileged information available only in the physics simulator distills its knowledge into a student policy that relies solely on proprioceptive sensor information. Simulation results across a wide range of dynamic viscosity changes ($10^{-7}$ to $10^{-2} m^2/s$) reveal that the DRL agent autonomously acquires non-sinusoidal adaptive gaits. These gaits improve propulsion velocity and transport efficiency, breaking the inherent limits of conventional sinusoidal and kinematic control. The findings establish that implicit environment inference via privileged information distillation is an effective approach to bypass the constraints of classical models under unpredictable fluid dynamics.
T. Kimoto, A. Yamano, Kohei Honda et al.· arXiv.org· 0 citations
Approaching a target position and holding station in flowing water is a fundamental and critical capability for robotic fish operating in natural aquatic environments. Despite decades of advances in enhancing swimming efficiency and maneuverability, this capability remains underdeveloped, largely owing to the insufficiently characterized, highly nonlinear fluid-structure interactions inherent to freely swimming robotic fish in flows. To bridge this gap, we propose the SWiFT framework, a Swimming With Flow Toolbox that enables the efficient exploration of an egocentric station-holding policy for a body and/or caudal fin (BCF) robotic fish in unknown and turbulent background flows via reinforcement learning (RL). Our SWiFT integrates a free-swimming flow-tank experimental setup with a highly efficient, physically consistent computational fluid dynamics (CFD)-based simulator and a systematic sim-to-real transfer pipeline. The resulting policy achieves substantial improvements over state-of-the-art methods across all metrics, most notably root-mean-square error (RMSE) of distance. Furthermore, we validated that egocentric feedback alone, without any explicit flow sensing, enables station-holding in unknown turbulent flows, closely mirroring the biological phenomenon of rheotaxis. Accordingly, the success of this egocentric station-holding policy not only advances robotic fish control toward real-world deployment, but also highlights SWiFT's promise as a foundation for tackling complex swimming tasks for underwater robots.
Xiaozhu Lin, Xuejiao Huang, Hongru Dai et al.· arXiv.org· 0 citations
This paper presents the development and implementation of an end-to-end control framework for a quadruped walking robot based on deep reinforcement learning. The primary objective of the study is to design and verify a control system capable of autonomously generating locomotion strategies. A model of the walking robot was developed using the Simscape Multibody toolbox, providing a physics-based simulation environment for training and evaluation. The proposed control approach employs a deep reinforcement learning agent trained using the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm. The agent learns locomotion behaviors directly from interactions with the simulated environment, without relying on predefined gait trajectories or manually designed control laws. Through iterative training, the agent optimizes its policy to maximize a predefined reward function, enabling the robot to discover efficient and stable movement patterns. Simulation results demonstrate that the TD3-based approach is highly effective for continuous control tasks involving systems with complex nonlinear dynamics. The trained agent successfully learned locomotion strategies, including dynamic gaits with flight phases, highlighting the ability of reinforcement learning methods to handle naturally unstable behaviors that are difficult to design using classical control techniques.
Filip Połatyński, Paweł Skruch· International Conference on...· 0 citations
Deploying learned control policies on low-cost robotic platforms introduces transport latencies and noisy motor feedback that systematically widens the sim-to-real gap. The chasm of simulation to deployment in hardware lies in the delay of the actuator reaching the commanded position. On platforms such as the Mini Pupper 2, a measured>50 ms transport delay transforms the locomotion task from a standard Markov decision process into a partially observable one. In this paper, we take a biologically inspired approach of handling noisy and delayed feedback to close the sim-to-real gap, thereby expanding the capability of reinforcement learning on cost-constrained hardware. Using a low-cost quadrupedal hardware platform, we find that using a forward model of the average actuator delay, paired with a time-aware neural network results in robust locomotion. Additionally, our time-aware neural network learned a central pattern generator (CPG): a self-sustaining rhythmic gait that is robust to +320 ms latency perturbations, mirroring the CPGs found in the spinal cords of vertebrates. We posit that temporal self-organization may be a general strategy for cost-constrained locomotion.
Javier C. Weddington, Bence P. Ölveczky, S. Baccus· arXiv.org· 0 citations
An integrated framework that combines a vision module for landing point and time prediction with a direct position and time conditioned RL locomotion policy, instead of intermediate velocity commands is proposed, which mitigates perception latency during dynamic interception.
Yi-Dong Zhu, Zibo Dai, Tong-Ning Zhang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
