Exploring RGBD Models for Force Estimation in Surgical Manipulation of Small Intestine

Abstract

Abstract Gentle tissue handling is critical in gastrointestinal surgery, yet most laparoscopic and robot-assisted systems lack force feedback, requiring surgeons to rely on indirect haptics or visual cues to deduce the applied force. Vision-based force estimation using computer vision (CV) backbones with RGBD inputs offers a promising alternative to hardware force sensors, but is limited by the scarcity of supporting datasets. To address this data scarcity problem, we present a custommade silicone bowel phantom with mesentery and an RGBD stereo video-force dataset with synchronized robotic states. We investigate the performance of CV models for 3D force estimation during surgical manipulation of the small intestine with a baseline experiment of RGB and RGBD CV models. Three CV models (3D-ResNet-18, (2+1)D ResNet, and Video Swin Transformer) in RGB and RGBD input modes were evaluated under five-fold cross-validation. Video Swin Transformer achieved marginally better force estimation error with RGB and RGBD input modes yielding similar performance, while all models maintained high inference capability. The proposed dataset and baseline experiments demonstrate the feasibility of RGBD-based surgical force estimation and provide a foundation for future research on intraoperative assistance and transfer to ex-vivo and clinical settings.