Robotic Hand‐Arm Coordinated Skill Learning for Tool Manipulation Tasks

Abstract

ABSTRACT This paper focuses on the specific scenario of tool manipulation and presents a novel framework enabling robots to acquire tool manipulation skills through learning from human demonstrations. While there have been numerous studies on robotic manipulation skill learning, most of them fail to fully leverage the combined advantages of coordinated hand‐arm movements—such as in dynamic tool repositioning and complementary workspace utilisation. To tackle this issue, the framework presented in this paper captures visual and tactile sensory streams from human demonstrations—including synchronous camera images and tool interaction forces acquired through optical tactile sensing—while recording hand‐arm coordination strategies. A Diffusion Policy model is adopted to learn robust manipulation skills from the demonstration data. During task execution, the robot fuses real‐time visual and tactile feedback to dynamically coordinate hand and arm motions. This work innovatively introduces object‐level stiffness as a feature characterising hand manipulation actions, allowing the robot to exhibit human‐like adaptive capabilities during skill deployment. We validate the proposed method in two real‐world tasks: cucumber peeling and bottle squeezing. Results show the robot achieves reliable manipulation under complex constraints and unknown perturbations. Multimodal imitation learning with fused vision‐tactile feedback enables efficient hand‐arm coordination, improving the robustness and flexibility of robotic tool manipulation.