This work tries to address the gap in Continual Reinforcement Learning by proposing a continual RL framework based on Continual Backpropagation that is able, with only real-world data, to train a generalistic policy on a set of tracks and then fine- tune it within 15 minutes to outperform classical controllers.
Abstract
A key challenge in modern robotics is to adapt to changing environments, a challenge that is exacerbated when simulations cannot encompass every possible real-world configuration, and therefore Reinforcement Learning (RL) in the physical world becomes necessary. Continual Reinforcement Learning provides the tools to address this challenge; however, both the frameworks and the methods remain underexplored. Autonomous Racing and in particular the RoboRacer competition provide a testing ground for such methods, as learning to drive on a new track-floor combination with the least amount of new experience naturally frames a continual learning problem. This work tries to address this gap by proposing a continual RL framework based on Continual Backpropagation that is able, with only real-world data, to train a generalistic policy on a set of tracks and then fine- tune it within 15 minutes to outperform classical controllers. Furthermore, a comparison method based on offline RL is proposed, and a simulation analysis of the plasticity properties of the methods is conducted.
The Open Ant is presented: a physical variant of the commonly used Gymnasium Ant environment, along with a simulation, that demonstrates that competent walking policies can be learned from scratch in approximately one hour directly from the physical robot's experience for two substantially different RL algorithms: SARSA($\lambda$) and Soft Actor-Critic (SAC).
E. Lupu, Patrick Spieler, Khurram Javed et al.· arXiv.org· 0 citations
This work identifies that additional research is still required to claim the successful resolution of the robotic arm reach-avoid task using DRL, and presents a comprehensive benchmark for the reachavoid task that accurately captures real-world complexities without simplifications.
Jonas Weihing, Shahram Eivazi· arXiv.org· 0 citations
We present an online learning framework that enables a bimanual robot to acquire diverse juggling patterns directly on physical hardware within minutes, even with a significant sim2real gap. One of the most important lessons from this work is that a model, even when far from reality, can be extremely useful for learning. This motivates a central philosophy of our approach: learning should build upon the robot's current knowledge rather than replace it. Our regularized memory-based learning puts this principle into practice by learning a local model from accumulated experience while retaining the global prior model to extrapolate where experience is sparse. This enables efficient and stable online learning from each new experience without resorting to uninformed exploration over a vast space of possible behaviors. Equally important to continual on-robot learning is safety, allowing the robot to repeatedly practice and improve in the real world. We construct a mutually reachable set that allows safe transitions between successive throws and catches, without driving either arm into a state from which its next action would require violating the robot's joint or actuator limits. Together, these ideas enable a bimanual robot with multi-fingered hands and onboard vision to safely learn and compose five canonical three-ball juggling patterns, including cascade, tennis, half-shower, shower, and box, within less than 5 minutes of real-world interaction. More broadly, this work points toward robots that build upon imperfect prior knowledge and continually refine their behavior through their own real-world experience.
Taeyoon Lee, Chun-Peng Wang, C. Atkeson et al.· 0 citations
Objective: In the field of mobile robotics, autonomous navigation in dynamic environments is one of the most challenging tasks in these environments: traditional methods based on pre-mapping and geometric planning are not effective in these environments due to uncertainty, and reactive methods are lacking in foresight. The work in this thesis tackles these issues by developing and testing an end-to-end Deep Reinforcement Learning (DRL) framework for maples navigation. Methods: A Proximal Policy Optimization (PPO) agent is trained using observations from LiDAR and goal-relative inputs in a high-fidelity open-source simulator that has been domain randomized in order to improve generalization. The trained policy is then transferred to a physical TurtleBot3 platform, with a safety supervisor controlled fine-tuning process. Comprehensive evaluation is performed on five simulated test scenarios (S=5), with 50 episodes per test scenario (250 episodes in total), with real-world trials performed on 20 trials in two different physical settings: cluttered lab and pedestrian corridor. The proposed approach is compared with a DDPG agent, as well as a standard A*+DWA pipeline using paired t-tests (α=0.05) and two-proportion z-tests, with statistical significance confirmed. Results: In simulation, the PPO agent has a success rate of 94% and a normalized path length of 1.18, both of which are significantly higher than those of A+DWA, which are 76% and 1.32 respectively (p<0.001 for both metrics); the agent takes 28.3 s to complete the task, which is significantly faster than A+DWA's 41.2 s (p<0.001). The success rate in the real-time laboratory test is 90%, and the inference time is 8.5ms/s. When the approach is used in structured corridor settings, the A+DWA baseline outperforms the PPO baseline with a 95% success rate vs. 85% for the PPO baseline, though. Novelty: The results of this study show that a PPO-based policy trained only in simulation and fine-tuned only on a small number of real-world tasks can compete with the classical and alternative DRL baselines in unknown and dynamic environments. The results form a feasible basis for the implementation of a learning-based navigation controller on low-cost mobile platforms and offer a fair comparison between the performance and limitations of learning-based navigation controllers and traditional ones.
Nabeel Muhamed, Khaleel Ali Khudhur· Jurnal Media Elektrik· 0 citations
This study proposes a new method, Wrench-Augmented Reinforcement Learning (WARL), which introduces a wrenche (force and torque) into the action space, and shows that introducing a wrench can encourage behaviors that do not sufficiently exploit the robot's physical embodiment.
Adapting a pretrained vision-language-action (VLA) policy to a new robot usually assumes embodiment-specific demonstrations. This assumption is especially restrictive for custom robots whose morphology differs strongly from the manipulators seen in large robot datasets. We study a harder setting: zero-demo embodiment alignment of OpenVLA-OFT on a cable-driven parallel robot (CDPR) with a simple gripper and a previously unseen control interface. Instead of supervised fine-tuning, we use reinforcement learning in simulation with dense geometric rewards computed from simulator state. The training is performed in two stages: a PPO stage for directional motion primitives, followed by GRPO continuation from the PPO checkpoint with an expanded instruction space that includes object-conditioned commands. On the four shared directional instructions, the average held-out success rate improves from 34.25\% after PPO to 53.50\% after PPO$\rightarrow$GRPO, with especially large gains on \texttt{move left} and \texttt{move backward}. In the GRPO stage we additionally introduce \texttt{move to} over eight target objects and obtain 39/400 = 9.75\% strict success, while qualitative rollouts frequently show correct target-directed approach behavior before late-stage instability. Compared with prior OpenVLA and OpenVLA-OFT results, which rely on demonstration datasets and mostly standard rigid-arm embodiments, our method uses no embodiment-specific dataset at all. The results do not yet establish robust manipulation, but they provide stronger evidence that RL-only bootstrapping can create the first usable language-conditioned controller for a genuinely novel embodiment.
Damir Nurtdinov, A. Kornaev, A. Maloletov· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
