Dynamic Object Tracking and Recovery for Language-Conditioned Mobile Manipulation

Abstract

Autonomous mobile manipulators operating in dynamic environments face significant challenges due to spatial variability, diverse task requirements, and the complexities of natural language understanding. Traditional systems often lack persistent perception, fail to adapt to open-vocabulary commands, and cannot recall spatial context from prior interactions. This paper presents a unified framework for mobile manipulation that integrates real-time visual tracking, language-driven task planning, and a retrieval-augmented memory system. The approach enables a robotic arm to maintain a moving target object within the center of the camera’s field of view, ensuring continuous visual contact. A large language model (LLM) interprets natural language commands and dynamically redefines target objects, facilitating flexible task execution across diverse categories. To support long-term reasoning, a Retrieval-Augmented Generation (RAG) module is adopted to archive the encountered object instances and their associated spatial contexts. Experiments conducted in real-world office environments demonstrate the system’s robustness in object tracking, context-aware manipulation, and language-conditioned generalization. The results confirm that the proposed system can reliably perform semantic search, object following, and adaptive manipulation in response to natural language commands.