A Distributed Conversational Framework for Human-Robot Collaborative Manipulation Using Local LLMS and VLMS

Abstract

This paper presents a distributed conversational framework for human-robot collaborative manipulation that integrates local language and vision-language models (VLMs) with a Robot Operating System 2 (ROS 2)-based execution stack. Language understanding, visual grounding, orchestration, and motion execution run as separate ROS 2 nodes, enabling flexible deployment across distributed hardware while maintaining a responsive control loop. From free-form user commands, the system generates structured action requests for pick, place, and handover. It uses a VLM to return image-space targets, which are converted into metric robot-frame goals using depth and calibration. A web dashboard exposes intermediate intent and grounding overlays (pixel, depth, and robot-frame) and requires explicit operator confirmation before any motion is executed. Experiments on a Franka FR3 platform evaluate task reliability and latency in a dice-based tabletop manipulation scenario under increasing scene ambiguity, comparing alternative LLM/VLM configurations. Code and full documentation are available at github.com/cogrob-tuni/franka-llm.