Towards Perception-Aware Shared Autonomy: A Multimodal Teleoperation Dataset for Fragile Object Manipulation

Abstract

Teleoperated fragile object manipulation is a critical capability in high-stakes domains such as remote hazardousenvironment operations and surgical robotics, where perceptual uncertainty can lead to irreversible damage. To understand the perception-control correlation, we present a dataset for analyzing human behavior in this setting, comprising 221 trials from 11 participants manipulating objects with 20 visually encoded fragility levels (50-1000 gf), indicating the maximum force can be exerted onto the object. The dataset includes synchronized multimodal observations: robot joint trajectories, gripper force signals, multi-view RGB video, users' perceived object fragility, confidence ratings, and trial outcomes. Our analysis reveals key behavior patterns: a 48% overall success rate under challenging fragile object constraints, generally under-estimating bias when perceiving object fragility, weak correlation between perception confidence and accuracy, substantial early-contact failures, and failure due to suboptimal control. We further show that precontact kinematics encode users' perception of object fragility while exhibiting strong user-specific baselines. We release this dataset as an open platform to support the development of perception-aware shared autonomy, where systems explicitly model operators' internal estimates of object properties rather than treating observable actions alone as ground truth. This enables research in user behavioral profiling, personalized user modeling, proactive intervention design, and adaptive robot control for damage-sensitive manipulation.