Vision Based Pick and Place of Randomly Stacked Jenga Blocks Using a Single RGB-D Sensor
Reliable manipulation of small, densely stacked objects remains a significant challenge due to severe occlusions and geometric ambiguities. When observed from a single RGB-D viewpoint, adjacent surfaces of featureless cuboids like Jenga blocks often merge in depth measurements, hindering accurate instance separation and pose estimation. This paper presents a unified perception and manipulation framework designed for the robotic rearrangement of randomly stacked Jenga blocks using a single Intel RealSense D435 sensor. Our approach integrates a perception pipeline based on heightmaps, which combines color extraction with geometric reasoning to robustly segment individual blocks and estimate poses directly compatible with grasp planning. To overcome the inherent limitations of sensing from a single view in dense clutter, we propose an iterative strategy of scanning, evaluating, and regrasping. When no immediate grasp is feasible, the system performs controlled actions of grasping and releasing to induce local reconfiguration, transforming cluttered states into graspable arrangements. Experimental results, conducted under conditions equivalent to the actual competition, demonstrate a 99.02% task success rate. The proposed framework ensures stable pick-and-place operations that inherently consider manipulation constraints, proving that robust task execution is achievable using a single RGB-D sensor.