A reinforcement learning-based framework for robotic grasp refinement, integrating keypoint-based object representations with a Deep Q-Network (DQN), is proposed, offering a scalable and adaptable solution for contact-rich manipulation tasks.
Abstract
Developing robots capable of understanding and manipulating objects requires compact, interpretable, and generalizable representations. This work proposes a reinforcement learning-based framework for robotic grasp refinement, integrating keypoint-based object representations with a Deep Q-Network (DQN). Using 2D overhead images captured in a simulated environment, a geometric-based algorithm generates initial grasp candidates, which are iteratively refined by the proposed framework, transforming failed grasps into successful ones. Experiments conducted on 300 objects from the Dex-Net dataset using a UR5 manipulator demonstrate the framework's effectiveness, achieving a 100% success rate on objects previously deemed ungraspable by geometrical methods. The framework's sim-to-real transferability is further validated through physical experiments on a Delta parallel robot, where a refined grasp successfully manipulates an object that was previously ungraspable. The findings underscore the effectiveness of reinforcement learning in addressing challenges in robotic grasping, offering a scalable and adaptable solution for contact-rich manipulation tasks.
Unlike rigid object grasping, manipulating articulated objects requires grasp poses that ensure both stability and actionability, that is, the ability to interact with an object’s joints. Therefore, we propose a deep learning-based grasp estimation model designed to enable robotic manipulation with articulated objects. The model first utilizes RGB-D images to predict graspable regions in the form of semantic heatmaps, identifying where to grasp. These regions are then combined with geometric features derived from point clouds through an attention-based semantic and geometric feature fusion module, ultimately to regress a manipulable 6-DoF grasp pose. To train and evaluate the proposed model, we constructed a custom dataset composed of RGB-D images and 6-DoF grasp labels collected from real-world articulated objects such as doors. Real-world robotic experiments demonstrated the effectiveness of the proposed model in producing stable and actionable grasps for articulated-object manipulation. Furthermore, ablation studies showed that incorporating the attention-based semantic and geometric feature fusion (ASGF) module improved the grasp success rate in the evaluated setting.
Dongwoo Lee, Yeongmin Kim, Seong-Bo Jo et al.· IEEE Access· 0 citations
Multifingered grasping is a crucial robotic skill, but current deep-learning grasp planners often struggle to generalize to new objects because they are trained on limited, object-specific datasets. We introduce a fundamentally different approach, grounded in the observation that the gripper and the object share identical surface geometry at their mutual contact points. We propose GOAG: Generative and Object-Agnostic Grasp Planner for Dexterous Robotic Manipulation, a novel deep generative model that learns a compact latent representation of a specific gripper's contact surface distribution, enabling the efficient sampling of valid grasp configurations without relying on object-specific training data. We show that by introducing object features only at inference time, our model can effectively retrieve admissible contact areas that are compatible with the gripper's capabilities. We validate our approach through extensive experiments on established grasp protocols in both simulated and real-world scenarios, demonstrating its effectiveness with different grippers from the literature. Our method delivers state-of-the-art results on the objects from the MultiDex dataset, achieving an average success rate of 86.93%. It offers significantly faster processing when generating numerous grasps, while matching the performance of leading approaches specifically trained on this dataset. Unlike these methods, our approach does not rely on object-specific training data, highlighting the advantages of object-agnostic learning. It effectively addresses the generalization challenges faced by traditional data-driven grasp planners. Code and videos are available on our project website https://cea-list.github.io/goagweb/ .
Julien Mérand, Boris Meden, Mathieu Grossard et al.· 1 citation
In complex backgrounds and under severe occlusions, the accuracy of vision-based robotic grasping pose estimation decreases significantly, further making objects difficult to manipulate and grasp. This paper proposes an iterative closed-loop optimization framework that deeply couples SAM with FoundationPose. The framework breaks through the open-loop logic bottleneck of “segmentation first, then estimation” found in traditional vision algorithms and constructs a mask correction mechanism based on rendered projection. By performing 3D rendering of the initially estimated 6D pose, a geometric prior mask of the object is generated and then fed back into SAM’s prompt encoder, thereby guiding the model to achieve pixel-level refinement of the target’s boundary in the next perception cycle. Meanwhile, to overcome the blind spots of a single metric, the framework designs a multi-dimensional confidence assessment module that integrates both the 2D image domain and the 3D geometric domain to comprehensively evaluate the reliability of the current pose. The SAM prior, the closed-loop iterative mechanism, and the multi-dimensional confidence assessment module work in synergy to form a complete optimization loop. In robustness experiments on pose estimation under cluttered and stacked scenes, the proposed method achieves an overall ADD-S recall rate of 91.7%, with the average translation and rotation errors reduced to as low as 3.5 mm and 2.1°. In 200 real-world robotic grasping verification trials, the overall grasping success rate reaches 96.5%. These experimental results demonstrate the effectiveness and enhanced robustness of the proposed closed-loop optimization framework in the tested unstructured environments.
Zhiqiang Gao, Mengqi Li, Huihui Bai et al.· Italian National Conference...· 0 citations
Neural-network-based grasp detection has achieved remarkable success in robotic manipulation due to its efficiency and generalization ability. However, detected poses are often not optimized, leading to undesired object motion or collisions during physical execution. This paper proposes a motion-aware refinement framework that minimizes estimated object motion while enforcing collision avoidance. The seven-dimensional pose is decomposed into approach direction, engagement depth, planar projection, and gripper opening width, enabling efficient and interpretable optimization in lower-dimensional subspaces. To evaluate grasp stability beyond conventional success metrics, we introduce the observed success rate (OSR) together with quantitative motion measurements including translation, rotation, and tilt. Real-robot experiments show that, for high-profile objects, the full pipeline improves the measured success rate (MSR) from 93.33% to 100% and OSR from 83.33% to 97.78%. It also reduces the mean translation from <inline-formula> <tex-math notation="LaTeX">$6.099{\,}mm$ </tex-math></inline-formula> to <inline-formula> <tex-math notation="LaTeX">$2.684{\,}mm$ </tex-math></inline-formula>, rotation from 3.732° to 1.344°, and tilt from <inline-formula> <tex-math notation="LaTeX">$4.417{\,}mm$ </tex-math></inline-formula> to <inline-formula> <tex-math notation="LaTeX">$1.313{\,}mm$ </tex-math></inline-formula>, while requiring <inline-formula> <tex-math notation="LaTeX">$0.82\pm 0.42{\,}s$ </tex-math></inline-formula> on average. For low-profile objects that cannot be detected by the baseline point-cloud-based planner, the full pipeline achieves 100% MSR and OSR.
Tian Tan, Redwan Alqasemi, R. Dubey· IEEE Access· 0 citations
Dual-arm robotic grasping is essential for manipulating large, heavy, and geometrically complex objects that cannot be reliably handled using a single manipulator. These large objects often contain only sparse graspable regions determined by local geometric properties such as thickness, edge structure, and gripper clearance. Prior bimanual grasping methods assume access to a full point cloud of the object which inherently contains this geometric information, but may not be accessible in real scenarios. This work proposes PartialBiGrasp, a dual-arm grasp generation framework that operates directly on partial point cloud observations. Our model learns geometric features implicitly through convolutional occupancy networks, enabling local reasoning about graspability, collision-free contact regions, and object thickness. We leverage this understanding to generate force-closure compliant grasp pairs, which are further refined using a sampling-based optimization to correct for ambiguity caused by incomplete geometry. We evaluate our approach using analytical force-closure metrics, large-scale simulation experiments, and real-world robot evaluations on noisy partial point clouds of novel objects, demonstrating robust and physically stable dual-arm grasp generation.
From feet to fingertips — we are teaching robots intelligent whole-body control, fine dexterity, and teamwork to complete a broad range of complex tasks.
