A Novel Dataset and Practical Deployment for Gesture-Based Human-Collaborative Robot Interaction

Abstract

Collaborative robot or COBOT has emerged as a key advancement in robotics, significantly transforming industrial automation. Consequently, real-time gesture recognition for human-COBOT interaction has recently garnered significant attention. However, both datasets designed for practical robot control scenarios and end-to-end deployment frameworks with seamless robot communication remain underexplored. To address these gaps, this paper makes two main contributions. First, we introduce CoboGestureV2, a novel dataset of 2362 gesture samples across 16 classes, specifically designed for continuous gesturebased COBOT control in practical workflows, covering both static and dynamic gestures including movement and rotation commands. Second, we present an end-to-end real-time gesture recognition framework deployed on an edge device (Jetson AGX Orin) and integrated with an MQTT-based communication protocol that enables lightweight, scalable message exchange between the recognition module, a coordination block, and the robot controller. On the proposed dataset, the framework achieves a frame-wise accuracy of 92.60%, edit score of 88.85% and TAL score of $\mathbf{7 1. 8 2} \boldsymbol{\%}$ in an offline setting, and $\mathbf{8 3. 6 5 \%}, \mathbf{7 7. 9 4 \%}$ and 48.93% for the respective metrics with a recognition latency of 0.87 seconds in a real-time online deployment.