HydroGym is introduced, a solver-independent reinforcement learning platform providing more than 60 validated, openly available flow control environments spanning from canonical laminar flows to complex turbulent flows, with systematic progression in the Reynolds number up to Re = 4 × 105, and Mach number variations in two and three dimensions.
Abstract
Effective control of fluid flows is critical across transportation, energy and medicine, where it can increase lift, reduce drag, enhance mixing and attenuate noise1-3. Yet fluids are notoriously difficult to control because they involve high-dimensional, nonlinear and multiscale dynamics that resist conventional approaches4-6. Reinforcement learning has driven remarkable progress in fields such as protein folding and complex games, which have shared benchmarks and standardized environments7-10. Fluid dynamics has lacked such infrastructure, so each controller is typically tuned to a single geometry and operating condition, making progress difficult to accumulate, transfer and compare11-13. Here we introduce HydroGym, a solver-independent reinforcement learning platform providing more than 60 validated, openly available flow control environments spanning from canonical laminar flows to complex turbulent flows, with systematic progression in the Reynolds number up to Re = 4 × 105, and Mach number variations in two and three dimensions. Across these environments, agents repeatedly discover robust control principles, including boundary layer manipulation, disruption of acoustic feedback and reorganization of turbulent wakes. Critically, we demonstrate a proof of concept for zero-shot transfer, in which agents that are trained exclusively in inexpensive surrogate environments are deployed to challenging real-world scenarios such as a three-dimensional wing section. We achieve a 38% reduction in local skin friction while reducing exploration costs by four orders of magnitude compared with direct on-wing optimization. As this transfer exploits shared near-wall physics, the breadth of generalization remains open, suggesting a new pathway for research toward policy generalization across computationally prohibitive simulation environments. By offering a common, extensible foundation for reproducible research, HydroGym moves flow control from isolated case studies toward a cohesive community effort.
Abstract Content of image described in text. Vertical-axis wind turbines (VAWTs) have emerged as natural complements to their horizontal counterparts but suffer from performance limitations associated with complex aerodynamics, in particular dynamic stall. To address this challenge, we employ deep reinforcement learning to identify optimal blade pitching policies. A framework based on the twin-delayed deep deterministic policy gradient algorithm is coupled with a validated numerical simulation of a two-bladed VAWT, aimed at maximising power generation. The agent receives only onboard measurements of blade pressure and position, rendering the problem partially observable. From these inputs, it learns policies for three distinct flow cases: one uniform flow and two shear flows. The resulting policies outperformed the optimum fixed-pitch setting within 100 turbine revolutions across all cases. Ultimately, the agent found cycle-averaged power coefficients that are 3.1–5.6 times larger than those of a blade fixed at zero degrees, while simultaneously reducing variations in blade loads. Improvements originate equally from both halves of turbine rotation, despite the latter-half’s low absolute power contribution and depleted flow energy. Through flow field analysis using the power partitioning method, the underlying mechanisms responsible for power production are revealed. The presence of a rotational flow around the airfoil leading edge dominates. Its net power contribution is determined by the rotational intensity and the blade’s kinematics. Kinematics also play a central role in mitigating dynamic stall. Flow separation is not intrinsically detrimental, as proper blade positioning can suppress its harmful effects.
Jarno Platenburg, Brice Martin, Thierry Jardin et al.· Journal of Fluid Mechanics· 0 citations
Reinforcement learning (RL) has recently emerged as a promising feedback control strategy for nonlinear and complex dynamical systems. However, RL algorithms are sample inefficient and require a large number of interaction with the environment to synthesize optimal control strategies. Consequently, applications of RL are typically limited to sparse sensors and actuators due to the curse of dimensionality entailed by the exploration-exploitation dilemma in high-dimensional spaces. In this work, we bridge RL and traditional optimal control for dynamical system with a novel Physics-EnhAnced Reinforcement Learning (PEARL) paradigm tailored to the control of high-dimensional and parametric dynamical systems, exploiting the differentibility of their dynamics. Specifically, PEARL employs an actor-adjoint algorithm that leverages automatic differentiation to compute policy gradients over short horizons and adjoint-based sensitivities of future returns approximated via neural networks, significantly reducing the number of environment interactions, while mitigating long-term gradient instabilities. Through two challenging parametric navigation problems in unsteady flows, we show that PEARL (i) effectively exploits differentiable environments to outperform state-of-the-art RL algorithms, (ii) is sample efficient, thanks to the physics-guided policy learning, (iii) generalizes across multiple scenarios, which is crucial when dealing with parametric systems, and (iv) enables scaling RL to high-dimensional state and action spaces, without requiring low-dimensional state representations or multi-agent strategies.
Matteo Tomasetto, Nicolò Botteghi, Gabriele Bruni et al.· 0 citations
This work investigates Deep Reinforcement Learning (DRL) as a tool for model-free closed-loop active separation control in a fully turbulent wind tunnel flow over a one-sided diffuser. The agent controls an array of magnetic valves (on/off) that eject compressed air into the boundary layer, while the environmental state is reduced to the signal from a single wall-shear-stress sensor placed near the natural transitory detachment point. The control law is learned in real time using Proximal Policy Optimization. Compared to the standard learning design based on the weighted sum of all rewards following an action, we demonstrate that a horizon aligned with the convective time of the flow leads to faster convergence and a more robust control strategy. The resulting control law corresponds to a low-duty-cycle actuation pattern that yields a forward-flow fraction of approximately $53\%$. This compares favorably with conventional and optimized periodic open-loop control ($\sim 40\%$ and $\sim 51\%$, respectively). The findings of this article indicate that, when embedded into an online experiment, DRL represents an efficient tool to identify robust and interpretable active separation control strategies.
Sofia Avdiiv, Andre Weiner, B. Steinfurth· 0 citations
Closed-loop wall control learnt by multi-agent reinforcement learning can lower skin-friction drag in turbulent channels, but these gradient-based policies are trained on small periodic boxes and exhibit reduced performance when carried over to a larger domain. We recently showed that such policies are also prone to saturated bang-bang actuations that collapse into standing streamwise waves whose scale is set by the computational box rather than by the near-wall cycle, and proposed architectural fixes that avoid these degeneracies. Here, we employ Evolution Strategy (ES) to optimise a recurrent closed-loop controller directly on a large turbulent channel at $\mathit{Re}_{\tau}\simeq180$, evaluating policy performance over full flow episodes using an energy-aware criterion and processing candidate policies in parallel. To our knowledge, this is the first application of an evolution strategy to the control of a turbulent flow. The ES controller reduces the skin friction by about $26\%$, exceeding the gradient-based multi-agent controller of Cavallazzi et al. (2026), GRU-MARL, trained on a minimal box ($17\%$), and marginally exceeding classic opposition control (OC, $22\%$). A wall-normal decomposition of the friction, Reynolds-stress profiles and anisotropy invariants show that the ES and opposition-controlled flows follow separate trajectories through the buffer layer, reaching comparable drag reduction by different reorganisations of the near-wall turbulence. In particular, the ES actuation correlates predominantly with the streamwise velocity fluctuations rather than with the wall-normal velocity that classical OC targets.
Giorgio Maria Cavallazzi, Miguel Pérez Cuadrado, Alfredo Pinelli· 0 citations
Abstract Content of image described in text. This study investigates the application of deep reinforcement learning (DRL) for active flow control in a three-dimensional NACA0012 wing section with periodic spanwise boundary conditions at low Reynolds number ( italic Re Subscript c Baseline equals 1000 Rec=1000
$ \textit{Re}_c=1000$
) and high angle of attack ( AoA equals 20 Superscript ring AoA=20∘
$\text{AoA}=20^\circ$
), where the flow exhibits massive separation, strong vortex shedding and a chaotic three-dimensional wake. The baseline configuration is validated against previous numerical studies, showing the accuracy of the computational set-up. Two DRL control policies are trained with different reward functions, leading to fundamentally different control strategies. The DRL set-up uses a multi-agent reinforcement learning framework to coordinate distributed actuators along the wing span. In the first case study, when the reward prioritises drag reduction, agents delay the leading-edge separation and stabilise the wake, reducing drag by 21.1 percent sign 21.1%
$21.1\,\%$
and lift fluctuations by 54.1 percent sign 54.1%
$54.1\,\%$
while maintaining baseline lift. In the second case, when the reward targets lift enhancement instead, agents synchronise actuations with vortex shedding, forming leading-edge instabilities that increase lift by 42.3 percent sign 42.3%
$42.3\,\%$
, albeit at the cost of higher fluctuations and slightly increased drag. A comparison with classical open-loop control further demonstrates that DRL autonomously determines physically meaningful and interpretable strategies consistent with established mechanisms in the literature. To the best of the authors’ knowledge, this is the first study in which a fully three-dimensional computational fluid dynamics simulation has been employed to train a DRL model for active flow control in wings, suggesting promising directions for extending DRL-based strategies to higher Reynolds numbers and more complex wing configurations where prior physical knowledge may be limited.
R. Montalà, B. Font, Pol Suárez et al.· Journal of Fluid Mechanics· 0 citations
Autonomous robotic navigation in nonstationary time-varying fluid flows remains a fundamental challenge due to partial observability and the unpredictability of realistic environments. While classical optimal control frameworks employed in robotics require unrealistic a-priori global flow knowledge, biological systems are able to navigate successfully by exploiting localized sensory cues. In this work we present a reinforcement learning approach using the TD3 algorithm to train autonomous agents to reach arbitrary targets within a parametric, chaotic double-gyre flow. To investigate optimal sensory mechanisms, we evaluate five bio-inspired observation strategies based on relative position, local velocity or local vorticity measures, and short-term memory variants. Additionally, we analyze the impact of providing agents with explicit global flow parameters. Numerical results demonstrate that an agent that is able to sense and remember a set number of flow velocity measures achieves the highest performance. The experiments reveal a trade-off in sensor utility: velocity-aware agents optimize energy efficiency, whereas vorticity sensors provide superior structural mapping and achieve better target proximity. Incorporating explicit global flow parameters is shown to decrease navigation performance. This behavior suggests that reinforcement learning-based autonomous systems develop more robust and general policies when restricted to implicit flow representations. The presented results offer insights for improving the transition of bio-inspired robotic navigation from simulation to real-world environments.
Andrea Braghin, Nicolò Botteghi, Matteo Tomasetto et al.· 0 citations