CoMuRoS - An LLM-based generalizable hierarchical task planning and execution framework for heterogeneous robot teams with event-driven re-planning

Abstract

Heterogeneous multi-robot teams require systems that can interpret natural-language goals, allocate tasks, and adapt to unexpected events. We developed CoMuRoS (Collaborative Multi-Robot System), a generalizable hierarchical architecture combining a centralized task-manager LLM with decentralized robot-level LLMs for executable Python generation from primitive ROS2 skills. The task manager uses static planning rules and dynamic context, including task history, robot/task status, and detected events, while onboard perception using VLM/image processing classifies events as relevant or irrelevant and triggers replanning. Hardware experiments demonstrated recovery from disruptive events, filtering of irrelevant distractions, and coordinated transport with emergent human-robot cooperation, achieving success rates of 9/10 for collaborative object recovery, 8/8 for coordinated transport, and 5/5 for human-assisted recovery. Simulation studies demonstrated intention-aware replanning. A curated benchmark of 22 scenarios, 54 tasks, and around 20 robots evaluated task allocation, classification, IoU, executability, and correctness across multiple LLMs, with correctness up to 0.91 ± 0.053; a 20-scenario replanning benchmark achieved Correctness = 0.948 ± 0.034 using Grok 3. CoMuRoS enables runtime, event-driven replanning on physical robots and supports flexible multi-robot and human-robot collaboration across diverse scenarios.