A Robust Visual Grasping Method for Robots in Cluttered and Stacked Scenes

Abstract

In complex backgrounds and under severe occlusions, the accuracy of vision-based robotic grasping pose estimation decreases significantly, further making objects difficult to manipulate and grasp. This paper proposes an iterative closed-loop optimization framework that deeply couples SAM with FoundationPose. The framework breaks through the open-loop logic bottleneck of “segmentation first, then estimation” found in traditional vision algorithms and constructs a mask correction mechanism based on rendered projection. By performing 3D rendering of the initially estimated 6D pose, a geometric prior mask of the object is generated and then fed back into SAM’s prompt encoder, thereby guiding the model to achieve pixel-level refinement of the target’s boundary in the next perception cycle. Meanwhile, to overcome the blind spots of a single metric, the framework designs a multi-dimensional confidence assessment module that integrates both the 2D image domain and the 3D geometric domain to comprehensively evaluate the reliability of the current pose. The SAM prior, the closed-loop iterative mechanism, and the multi-dimensional confidence assessment module work in synergy to form a complete optimization loop. In robustness experiments on pose estimation under cluttered and stacked scenes, the proposed method achieves an overall ADD-S recall rate of 91.7%, with the average translation and rotation errors reduced to as low as 3.5 mm and 2.1°. In 200 real-world robotic grasping verification trials, the overall grasping success rate reaches 96.5%. These experimental results demonstrate the effectiveness and enhanced robustness of the proposed closed-loop optimization framework in the tested unstructured environments.