Aug 2026· Intelligent Service Robotics· Vol 19· 0 citations· 42 references
Computer Science
TL;DR
A modular perception pipeline for household manipulation tasks, with a focus on dishware handling in kitchen environments, that integrates open-vocabulary object detection, multi-view segmentation, instance-aware 3D reconstruction, and a 2D-3D feature fusion strategy for 6D pose estimation and grasp planning is presented.
Abstract
Deploying robots in everyday human environments requires perception systems that are both robust and adaptable to diverse, dynamic conditions. In this work, we present a modular perception pipeline for household manipulation tasks, with a focus on dishware handling in kitchen environments. The pipeline integrates open-vocabulary object detection, multi-view segmentation, instance-aware 3D reconstruction, and a 2D-3D feature fusion strategy for 6D pose estimation and grasp planning. Its modular design enables systematic substitution of multiple visual and geometric foundation models, allowing us to identify the best-performing configuration through extensive evaluation on a custom kitchen dataset. The best-performing configuration (LLMDet + SAMv2 + DINOv2 + GeoTransformer) achieves an ADI of 89.12% on the 20-scene kitchen benchmark with cluttered and occluded conditions. Furthermore, real-world demonstrations confirm that the best configuration can be deployed on physical robots without environment-specific retraining, successfully executing tasks such as sink-to-dishwasher transfer and cup stacking. It validates the adaptability and scalability of the pipeline and highlights its potential as a practical framework for household robotic systems. Our code and supplementary materials are available at https://raivlab.github.io/FM_kitchen/.
Bimanual manipulation policies require large and diverse training datasets, yet collecting demonstrations on physical robots is expensive and difficult to scale. Simulation can generate data efficiently, but existing pipelines typically operate within closed asset libraries and predefined scenes: adding a newly observed object or environment still requires substantial effort to reconstruct geometry, specify physical and semantic properties, annotate interactions, and integrate the result into executable tasks. We present RoboCousin, an extensible simulation-based data-generation platform that turns user-provided observations into reusable assets, scenes, and expert trajectories for bimanual manipulation. Built on RoboTwin~2.0, RoboCousin converts object images into simulation-ready assets with visual and collision geometry, semantic and physical metadata, and automatically generated grasp-contact candidates. It further constructs digital cousins that vary compatible objects, backgrounds, layouts, and language instructions while preserving task-relevant affordances and spatial relations. The same asset system supports tabletop and room-level scene construction, with collision-aware base control for interaction beyond a fixed workspace. We release RoboCousin-OBD, containing more than 3,000 annotated object instances and 50 background environments, and use RoboCousin to generate over one million expert trajectories across 50 tasks. Simulation and real-robot experiments show that the automatically generated interaction annotations are comparable to curated annotations, generated assets provide effective sim-to-real supervision, and tabletop cousins can improve transfer beyond training on a single reconstructed scene. RoboCousin therefore provides a practical path for expanding both the scale and coverage of synthetic bimanual manipulation data.
Jingbo Zhu, Jing-Yi Li, Liang-Liang Chen et al.· 0 citations
Deploying navigation systems at scale requires a recipe that minimizes sensor assumptions, generalizes across robot embodiments, and trains efficiently. Yet, today's best systems depend on depth sensors, multi-camera rigs, or pre-built maps, limiting the hardware they support and increasing deployment cost. We introduce Robostral Navigate, an 8B vision-language model built around this scalability objective. The model consumes only a stream of monocular RGB images - the most ubiquitous sensor across robotic platforms and predicts waypoints by pointing to the next target location in the current camera view. Operating purely in image space, rather than robot-specific coordinates, makes the policy naturally robust to changes in camera intrinsics and scene scale, enabling deployment across wheeled, legged, and aerial robots without recalibration. We generate 2.4 million trajectories across 350k simulated scenes to reduce the reliance on real-world data collection and scale easily. We further introduce a prefix-caching training recipe that packs entire episodes into single training sequences, reducing training tokens by 22x and cutting training time from months to days. A tree-based attention mask prevents conditioning on previous ground-truth actions, encouraging visually grounded action prediction, and reinforcement learning is used to further improve exploration and recovery capabilities. On the Room-to-Room and Room-Across-Room in Continuous Environments (R2R-CE and RxR-CE) benchmarks, Robostral Navigate sets a new state of the art. On R2R-CE, it achieves a 77.4% success rate, surpassing the best monocular method by 10.5 points and the strongest depth- or multi-camera system by 5.3 points despite using only a single RGB camera. On RxR-CE, it reaches 75.1% success rate, outperforming all monocular baselines.
Arjun Majumdar, Avinash Sooriyarachchi, Benjamin Tibi et al.· 1 citation· ⚡1
Open-vocabulary grasping on a quadruped manipulator requires more than recognizing the target object. The robot must also select a grasp pose that is both consistent with the task semantics and reliable to execute under body motion and viewpoint changes. In this paper, we present VLEG, an embodied vision-language grasping framework for quadruped manipulators that explicitly incorporates body motion into grasp decision making. Our method guides the robot to continuously adjust its body pose during approach and optimize local observations before grasping, thereby improving perception quality. For grasp decision making, instead of using a coarse single-stage filtering strategy, we design a multi-stage and multi-criteria grasp selection mechanism based on geometric grasp candidates. This mechanism jointly considers physical feasibility and task consistency. We implement the complete system on an onboard Jetson platform and conduct extensive real-world experiments on a quadruped robot equipped with a manipulator, covering tabletop, low-platform, ground-level, and outdoor raised-platform scenes. The results validate the deployability of VLEG in real-world quadruped manipulation scenarios, as well as its robust grasping ability and task-aware decision-making capability across the tested object categories.
Yu-Xing Ji, Fei Meng, Zishang Ji et al.· Journal of Physics, Conferen...· 0 citations
It has always been expected that robots can actively manipulate complex environments to fulfill human requirements. This process typically necessitates that the robot be equipped with the ability for embodied exploration and manipulation. To achieve this goal, in this paper, we propose to incorporate multi-source knowledge to enhance the ability of robotic embodied exploration and manipulation. Specifically, to eliminate the inherent biases in decision-making of large language models (LLMs), we introduce a multi-source knowledge fusion module to generate more reasonable exploration sequences. Notably, grasping detection plays a critical role in the process of robot manipulation. To achieve a better balance between the accuracy and efficiency of the grasping detection network, we design a two-branch feature fusion module with residual blocks to improve network performance. Conditioned on the aforementioned innovations, the robot is capable of actively exploring and manipulating in constrained environments to meet human requirements. Extensive experiments are conducted in both simulation and real-world environments. The results demonstrate the effectiveness and efficiency of our proposed framework.
Jin Liu, Kai Sun, Leibing Xiao et al.· Robotica (Cambridge. Print)· 0 citations
Industrial disassembly processes require robust and efficient perception systems capable of handling heterogeneous objects under real-world constraints. In automotive manufacturing and disassembly scenarios, components exhibit significant variability in geometry, size, symmetry, and placement, making it impractical to rely on a single, uniform object pose estimation strategy. At the same time, such environments impose strict requirements on the perception systems in terms of reliability, computational efficiency, and ease of deployment.This work presents an instance-based object perception pipeline with task-driven object pose estimation, developed within the SOPRANO EU project, for automated automotive door disassembly. The pipeline assumes a known set of object instances and depending on the objects’ geometric characteristics, the manipulation task requirements , the system performs (i) full 6D pose estimation, (ii) planar-constrained 3D localization via RGB-D lifting, or (iii) planar localization with structured multi-instance refinement and instance identification.The perception process consists of two stages: pose formulation selection and execution of the corresponding estimation method. Model-based 6D pose estimation using RGB data is applied to rigid objects, while RGB-D-based lifting of 2D detections is used for planar or quasi-planar elements. For structured arrangements of multiple instances of a single object type, such as screw arrays, a multi-instance matching strategy ensures consistent indexing and reduces ambiguity.The system is deployed as a modular perception service and validated in an automotive disassembly pilot. Experimental results demonstrate high accuracy across heterogeneous tasks, highlighting the benefits of aligning perception outputs with task-specific requirements.
Evangelos G. Sartinas, Athina Zacharia, Maria Pateraki· AHFE International· 0 citations
A reinforcement learning-based framework for robotic grasp refinement, integrating keypoint-based object representations with a Deep Q-Network (DQN), is proposed, offering a scalable and adaptable solution for contact-rich manipulation tasks.
Amir Arsalan Nematollahi, Shayan Ahmadi, M. T. Masouleh et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
