From Human Demonstrations to Robotic Actions: Real2Sim Verification of a Diffusion Policy for Autonomous Robotic Manipulation

Abstract

The Universal Manipulation Interface (UMI), originally developed by the Robotics and Embodied AI Lab at Stanford University, has demonstrated remarkable effectiveness for training manipulation policies for terrestrial robotic manipulators using imitation and diffusion based learning techniques. The long term objective of our research is to extend this technology to space robotic applications. There are numerous challenges largely unexplored, including harsh environmental conditions, stringent power and computational constraints, communication latency, and very limited opportunities for data collection and validation. This paper presents the first step toward achieving that goal by designing a similar gripper and testing it in both simulation and experiment with a Franka Emika robotic arm in our lab setting. We reproduce the data collection and diffusion policy training pipeline on commodity hardware, with a ViT-B/16 Vision Transformer serving as the policy’s vision backbone, and reconstruct a Franka Emika manipulator equipped with a custom electric gripper inside NVIDIA Isaac Sim, using the Lula inverse kinematics solver to perform kinematic control from the policy generated end effector commands. We identify and formalize the coordinate and action frame transformations required to transfer a policy trained on handheld demonstrations onto a simulated embodiment, and show that the simulated controller tracks the commanded trajectories to subcentimeter accuracy. We further report pick and place rollout statistics across randomized object configurations and identify the visual domain gap between rendered and real observations as the dominant remaining barrier to transfer. These results helped us understand the UMI framework and established a solid foundation for us to move forward toward free floating microgravity manipulation and autonomous dual arm object handover.