Dec 2026· IEEE Robotics and Automation Letters· Vol 11, pp. 13821-13828· 0 citations· 47 references
TL;DR
An articulation-driven digital twin augmentation approach that couples reconstruction, online articulation-state estimation, and observation generation within a shared 3D Gaussian Splatting (3DGS) model to demonstrate accurate articulation tracking and improved viewpoint generalization of 3D Diffusion Policy.
Abstract
Single-view 3D visuomotor policies based on point-cloud observations, such as 3D Diffusion Policy, often suffer from limited viewpoint generalization due to partial observability and occlusion. This is particularly challenging for articulated objects whose internal motion induces geometry changes. To address this problem, we propose an articulation-driven digital twin augmentation approach that couples reconstruction, online articulation-state estimation, and observation generation within a shared 3D Gaussian Splatting (3DGS) model. Offline, we reconstruct an articulation-aware 3DGS part model and parameterize the articulation. Online, we optimize joint states through differentiable rendering conditioned on the base pose, then transform pre-sampled part point clouds using the estimated configuration. Fusing the object point cloud with robot geometry enables real-time augmentation from partial to global observations using single-camera input during execution. Experiments on the complete pipeline in simulation and the real world demonstrate accurate articulation tracking and improved viewpoint generalization of 3D Diffusion Policy.
It is shown that a flexible and expressive transformer, PointZero, outperforms prior methods on the same data and demonstrates the utility of the pre-training objective by post-training PointZero for two downstream applications: action-conditioned 3D dynamics prediction and imitation learning.
B. P. Duisterhof, Kai-Feng Zhang, Adam Hung et al.· arXiv.org· 6 citations· ⚡1
Video demonstrations offer a scalable alternative to costly robot data for learning manipulation, yet existing reconstruction-based approaches often rely on constrained camera viewpoints or human-to-robot retargeting, while the reconstructed trajectories are difficult to adapt to new objects configurations without dist...
Hang Li, Ming-Xin Zhang, Zi-Han Wu et al.· arXiv.org· 1 citation
This work studies the effect of extrinsic reliability and pose conditioning on feed-forward 3DGS, and proposes a calibration-aware pipeline that anchors reconstructed scenes to the robot's metric workspace and presents a renderable, metric-pose-anchored dataset with scene-level reliability information for robot manipul...
Online 3D Gaussian Splatting (3DGS) is emerging as a practical way to build a representation for robotics applications. However, deploying it on real-world platforms exposes multiple challenges. Rapid maneuvers of a robot induce motion-blurred images, which corrupt the reconstructed geometry and cause camera pose drift...
Sang-Woo Jung, Dian Deng, Jia-Hui Fu et al.· 0 citations
This work introduces a calibration pipeline that combines learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics, and trains DepthWorld, a Stable Video Diffusion-based world model that jointly...
Jai Bardhan, Josef Sivic, Vladimír Petrík· 2 citations
Feed-forward 3D reconstruction turns an uncalibrated monocular stream into a dense point cloud in a single inference pass, orders of magnitude faster than classical structure-from-motion (e.g., COLMAP), and reliable on the weakly textured, repetitive surfaces where keypoint matching fails outright. However, raw point c...
Quan-Yun Wu, Kyle Gao, Daisy Li et al.· 0 citations
Related blog posts
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT scientists developed a training method to generate models that simulate the effects of human activities on climate much more quickly than other approaches.