Vision-Guided Arm–Hand Decoupled Motion Retargeting Using Graph Networks for Cross-Platform Robots

Abstract

Motion retargeting enables robotic systems to replicate human motion for complex tasks. However, existing approaches often depend on costly motion capture systems and struggle with scale discrepancies between hands and arms, leading to reduced precision. They also face challenges such as high computational cost, local minima, and poor generalization across different robot platforms. In this paper, we propose a novel vision-guided motion retargeting framework and approach to address these limitations. Our framework employs an arm-hand decoupled architecture, processing hands and arms separately to effectively mitigate scale discrepancies, thereby enhancing precision. The proposed retargeting method GNNRetarget integrates a graph encoder network to generate informative initial embeddings that are subsequently refined in the latent space. This strategy significantly reduces complexity while circumventing local minima issues. Crucially, by modeling human poses and robot kinematics on a unified graph, our method does not require manually annotated human–robot correspondences or pre-collected paired robot-motion datasets; instead, the correspondences are constructed automatically through kinematic graph mapping and refined by the objective function, enabling broad applicability across diverse robots. Experiments successfully replicate human motions and validate the feasibility and accuracy of our retargeting framework in simulated settings with RMC-DA, YuMi, and Unitree H1, as well as real-world deployment on RMC-DA, highlighting its practical value. Project can be found at: https://github.io/GNNRetarget/ Note to Practitioners—This study addresses the practical challenge of retargeting human motion to robotic platforms using only visual input, reducing reliance on costly motion capture systems and platform-specific rules. For applications such as teleoperation, service robotics, and collaborative manufacturing, precise and generalizable imitation is essential. To this end, we develop a vision-guided retargeting framework that decouples hand and arm movements into separate streams to mitigate scale discrepancies and improve accuracy. Furthermore, human poses and robot structures are represented as graphs, enabling retargeting across heterogeneous robot platforms without manually annotated human–robot correspondences or pre-collected paired robot-motion data. A latent-space optimization module refines motion representations efficiently and avoids convergence to local minima, enabling deployment across diverse robots with minimal adaptation. Nonetheless, performance may be affected by severe occlusion or low illumination; future work will incorporate depth and tactile sensing to improve robustness in real-world deployments.