GUIDER: Evaluating Goal-Free Human Intent Inference for Teleoperated Manipulation on Real-Robot Data

Knowledge-augmented embodied exploration for robotic grasping in constrained environments

It has always been expected that robots can actively manipulate complex environments to fulfill human requirements. This process typically necessitates that the robot be equipped with the ability for embodied exploration and manipulation. To achieve this goal, in this paper, we propose to incorporate multi-source knowledge to enhance the ability of robotic embodied exploration and manipulation. Specifically, to eliminate the inherent biases in decision-making of large language models (LLMs), we introduce a multi-source knowledge fusion module to generate more reasonable exploration sequences. Notably, grasping detection plays a critical role in the process of robot manipulation. To achieve a better balance between the accuracy and efficiency of the grasping detection network, we design a two-branch feature fusion module with residual blocks to improve network performance. Conditioned on the aforementioned innovations, the robot is capable of actively exploring and manipulating in constrained environments to meet human requirements. Extensive experiments are conducted in both simulation and real-world environments. The results demonstrate the effectiveness and efficiency of our proposed framework.

From Human Demonstrations to Robotic Actions: Real2Sim Verification of a Diffusion Policy for Autonomous Robotic Manipulation

The Universal Manipulation Interface (UMI), originally developed by the Robotics and Embodied AI Lab at Stanford University, has demonstrated remarkable effectiveness for training manipulation policies for terrestrial robotic manipulators using imitation and diffusion based learning techniques. The long term objective of our research is to extend this technology to space robotic applications. There are numerous challenges largely unexplored, including harsh environmental conditions, stringent power and computational constraints, communication latency, and very limited opportunities for data collection and validation. This paper presents the first step toward achieving that goal by designing a similar gripper and testing it in both simulation and experiment with a Franka Emika robotic arm in our lab setting. We reproduce the data collection and diffusion policy training pipeline on commodity hardware, with a ViT-B/16 Vision Transformer serving as the policy’s vision backbone, and reconstruct a Franka Emika manipulator equipped with a custom electric gripper inside NVIDIA Isaac Sim, using the Lula inverse kinematics solver to perform kinematic control from the policy generated end effector commands. We identify and formalize the coordinate and action frame transformations required to transfer a policy trained on handheld demonstrations onto a simulated embodiment, and show that the simulated controller tracks the commanded trajectories to subcentimeter accuracy. We further report pick and place rollout statistics across randomized object configurations and identify the visual domain gap between rendered and real observations as the dominant remaining barrier to transfer. These results helped us understand the UMI framework and established a solid foundation for us to move forward toward free floating microgravity manipulation and autonomous dual arm object handover.

ActivAsk: Free-Energy-Guided Clarification for Robotic Grasping Under Ambiguous Instructions

Service robots often receive natural language instructions in changing workspaces where multiple visible objects may match one description. Relying on detector confidence, random selection, or direct vision–language model (VLM) prediction can lead to a wrong action. This paper presents ActivAsk, a zero-shot framework for resolving referential ambiguity before robotic grasping. ActivAsk constructs open-vocabulary candidates from red-green-blue-depth (RGB-D) input, asks candidate-grounded yes/no questions when needed, updates the candidate state from the user’s answer, and grasps after target resolution. It selects among VLM-proposed candidate partitions using an expected free energy (EFE) criterion motivated by active inference; with neutral response preferences, this reduces to information gain over candidate partitions. Offline experiments showed that interactive clarification improved target accuracy from about 53–54% for noninteractive baselines to about 90–92%. ActivAsk matched the best interactive accuracy (92.13%) while asking 15.47–19.71% fewer questions on asked trials and 21.43–23.88% fewer for ambiguous instructions. In online real robot experiments, ActivAsk achieved 92.98% target selection accuracy and 87.72% full correct object grasp success; unresolved or wrong targets were not physically executed after operator-controlled verification and were counted as task failures.

Agent as Policy for Robotic Manipulation

We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, dice flipping, targeted throwing, and bimanual towel folding. Across assembly, block construction, and dice flipping, AGP succeeds in at least eight of ten trials for each evaluated task configuration. We further study efficiency through task experience accumulation and find that reusing saved procedures and programs shortens execution time across repeated trials. These findings support a path for general-purpose agents to act as robot policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.

Gemini Robotics 2: Safety Evaluations

HODAgent: Towards On-Demand, Responsive Humanoids for Physical World Human Interaction

Results show a unified System-2 agent enables adaptive humanoid service across simulation and reality.