GAS-Robo: Converting Scene Into a Grid-Action Space for LLM-Driven Open-Ended Robotic Manipulation

Abstract

Generalizing to out-of-distribution scenarios remains a major challenge for traditional robotic manipulation methods trained on closed datasets. Recent approaches leveraging foundation models have significantly improved zero-shot capabilities by utilizing vision-language models, yet many of these methods treat the foundation model as a high-level decision-maker, which limits their adaptability. A key challenge is the gap between high-level decision-making and low-level control. To address the challenge, we propose GAS-Robo, an open-ended manipulation framework that bridges this gap by utilizing a Grid-Action Space (GAS). GAS provides both semantic and spatial information to large language models (LLMs) and enables the direct generation of low-level actions, instead of invoking predefined APIs, thereby enhancing flexibility in trajectory control. The framework comprises two key components: an Environment Filter, which generates a task-aware grid representation of the scene, and an LLM-based Planner, which produces primitive action sequences based on the grid. To improve spatial reasoning and interpretability, a Chain-of-Thought (CoT) mechanism is incorporated into the planner. We benchmark GAS-Robo in the RLBench simulation environment, demonstrating state-of-the-art performance. Furthermore, physical validation on a Franka robotic manipulator platform highlights GAS-Robo's superior generalization across diverse real-world tasks, validating its real-world applicability and robustness in handling diverse manipulation tasks.