A data-driven scoping review of 130 studies published between 2020 and 2026, following PRISMA-ScR guidelines, to systematically map the landscape of long-horizon RL for robotic manipulation and presents a gap atlas that identifies underexplored research directions across methodological and experimental dimensions.
Abstract
Long-horizon robotic manipulation presents fundamental challenges for reinforcement learning (RL), including multi-stage decision-making, delayed credit assignment, and the integration of perception, planning, and control over extended time scales. Despite rapid progress, the field remains fragmented across methodological paradigms, task domains, and evaluation practices. This paper presents a data-driven scoping review of 130 studies published between 2020 and 2026, following PRISMA-ScR guidelines, to systematically map the landscape of long-horizon RL for robotic manipulation. We introduce a design-space perspective defined by three structural axes–temporal abstraction, predictive modeling, and structural guidance–and four recurring architectural archetypes–hierarchical, model-based, hybrid, and end-to-end model-free or offline–and provide a descriptive quantitative analysis of task distributions, sensing modalities, action representations, and evaluation protocols. Our analysis reveals a strong concentration of research in simulation-based settings, block rearrangement tasks, and state-based control, alongside limited exploration of real-world-only learning, reset-free execution, dexterous and deformable manipulation, and multimodal perception. To synthesize these findings, we present a gap atlas that identifies underexplored research directions across methodological and experimental dimensions. Finally, we propose a set of standardized reporting recommendations to improve transparency, reproducibility, and comparability in future work. Together, this review provides a comprehensive evidence map of the field and outlines key directions for advancing long-horizon robotic learning toward more realistic and deployable systems.
A coherent map of the rapidly expanding landscape of visual RL is provided to provide researchers and practitioners with a coherent map of the rapidly expanding landscape of visual RL and to highlight promising directions for future inquiry.
Reinforcement learning (RL) has emerged as a promising approach for autonomous surgical robotic subtasks. Recent advances include deep reinforcement learning (DRL), imitation learning (IL), and vision–language–action (VLA) models. However, current evidence remains fragmented across simulation benchmarks, task-specific demonstrations, and limited clinical studies. Existing reviews primarily focus on RL algorithms, while the broader pathway from algorithm development to clinically deployable surgical autonomy has not been comprehensively synthesised. This PRISMA 2020-guided systematic review examines RL, IL, safe RL, simulation-to-real (sim-to-real) transfer, foundation models, VLA systems, and regulatory readiness in surgical robotics. We searched IEEE Xplore, PubMed/MEDLINE, Embase, Scopus, Web of Science, the Cochrane Library, ACM Digital Library, arXiv, and medRxiv for studies published between January 2015 and March 2026, with additional studies identified through backward citation tracing. Eligible studies proposed novel RL, imitation learning, or foundation-model approaches for surgical robotics with empirical validation in simulation or on physical robotic platforms. Two reviewers independently extracted data using a predefined coding scheme, and a third reviewer resolved disagreements. Owing to substantial heterogeneity in platforms, tasks, and outcome measures, a quantitative meta-analysis was not feasible; therefore, the evidence was synthesised narratively using a comparative framework. A total of 220 studies met the inclusion criteria, covering eleven active surgical RL platforms, seven paired sim-to-real studies, emerging foundation-model architectures, and three FDA-cleared robotic systems exhibiting Level 3 autonomy. Available comparative studies suggest that hierarchical approaches can outperform flat policies in long-horizon tasks, while language-conditioned models demonstrated promising multi-step surgical capabilities. Seven paired simulation-to-real studies were identified, encompassing tissue retraction, guidewire navigation, and surgical cutting tasks. Sim-to-real performance gaps varied substantially by task and metric, with success-rate gaps ranging from −10 to 50 percentage points (negative values indicating better real-world than simulated performance), while paired mean spatial errors differed by at most 0.61 mm. Most studies employed domain randomization or visual domain adaptation; hierarchical reinforcement learning demonstrated advantages over flat policies in multi-step surgical tasks. Explicit safety-constrained methods (CPO, CBF, and SER), formal verification, and regulatory-aligned evaluation were reported in fewer than 3% of applied studies. Most evidence remained simulation-based, with no reported autonomous RL execution in vivo in humans. Overall, RL-based surgical robotics appears mature at the simulation stage but remains preclinical for autonomous clinical deployment. Future progress requires stronger sim-to-real validation, multimodal safety-aware architectures, alignment with IEC 62304, ISO 14971, FDA guidance, and the EU AI Act, and open benchmarks that jointly evaluate performance, safety, and surgeon trust.
M. Shahid, Abdullah, Zulaikha Fatima et al.· Biomimetics· 0 citations
Modern manufacturing faces increasing demands for flexibility, customization, and productivity under dynamic conditions. Multi-robot systems offer a promising solution by enabling cooperative execution of complex tasks, such as assembly and cooperative manipulation. In this context, Multi-Agent Reinforcement Learning (MARL) has emerged as a promising paradigm to enhance coordination and adaptability in industrial settings. MARL enables multiple agents to learn and interact in shared environments to achieve common goals within complex and dynamic industrial processes. In this paper, a deep analysis of MARL applied to industrial multi-robot systems based on a systematic review is presented, with particular focus on cooperative manipulation tasks. Following PRISMA guidelines, we analyze a total of 30 articles published between 2016 and 2026, selected independently by two of the authors from an initial pool of 102 records retrieved from Scopus and Web of Science. These articles were used to address five key questions regarding MARL algorithms, control architectures, industrial applications and validation practices. These research questions seek to examine gaps and trends at the research level which are important for the development of multi-agent control technologies. This review shows a clear prevalence of model-free algorithms under Centralized Training with Decentralized Execution (CTDE) architectures, with validation mainly performed in simulation. Despite promising results and high potential for impact, critical gaps remain in scalability, reproducibility, and sim-to-real transfer, limiting real deployment in manufacturing environments. To address these challenges and fill current gaps, we outline actionable research directions, such as hybrid MARL approaches, standardized industrial benchmarks, digital twin pipelines, and safety-aware deployment strategies, to accelerate MARL adoption in industrial environments.
Francisco J. Huertos, Oihane Bañales, Pedro Álvarez et al.· Robotics· 0 citations
This work identifies that additional research is still required to claim the successful resolution of the robotic arm reach-avoid task using DRL, and presents a comprehensive benchmark for the reachavoid task that accurately captures real-world complexities without simplifications.
Jonas Weihing, Shahram Eivazi· arXiv.org· 0 citations
This work uses Sample-based Model Predictive Control entirely in simulation as an automated, rapidly tunable expert to generate massive offline datasets and validate the robustness of this sim-to-real framework by successfully deploying complex loco-manipulation skills across different morphologies.
Martin Schuck, Maks Sorokin, S. Manni et al.· 0 citations
Diffusion policies model multimodal robot action sequences, but behavioral cloning does not directly optimize task return. We present a structured scoping review of reinforcement learning for generative robot policies and a bounded state-based locomotion reproduction. Four documented routes yielded 178 records, 162 unique candidates, and an 84-study evidence map. Hierarchical rules distinguish 41 direct reward-driven studies from 32 adjacent robotic, eight alternative-generator, and three non-robotic studies; a five-axis taxonomy codes initialization/data, interaction regime, optimized object, credit assignment, and generator. Under a fixed-final evaluation protocol on the Datasets for Deep Data-Driven Reinforcement Learning (D4RL) 1.1 Hopper benchmark, five diffusion policy policy optimization (DPPO) fine-tuning seeds improved over their run-recorded behavior-cloning initializations by a mean of 1261.2 return, with a seed-level standard deviation of 125.5 and a 95% confidence interval of 1105.3–1417.1; the five runs link to two recorded behavior-cloning checkpoints. A Gaussian-policy control also improved after proximal policy optimization, so the gain was not diffusion-specific. A full-chain backpropagation adaptation exhibited clear seed-dependent variation, a matched action-divergence intervention did not establish causal critical timesteps, and reducing denoiser evaluations from 20 to 2 lowered A100 latency from 30.97 to 3.85 ms while substantially reducing normalized score. The experiments are limited to state-based locomotion and do not validate visual manipulation.
Shihan Sun, Yinlong Liu· Robotics· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
