Skip to content
#diffusion models Open access

3D Gaussian Digital Twins for View-Robust 3D Visuomotor Policy Learning on Articulated Objects

Dec 2026 · IEEE Robotics and Automation Letters · Vol 11, pp. 13821-13828 · 0 citations · 47 references

TL;DR

An articulation-driven digital twin augmentation approach that couples reconstruction, online articulation-state estimation, and observation generation within a shared 3D Gaussian Splatting (3DGS) model to demonstrate accurate articulation tracking and improved viewpoint generalization of 3D Diffusion Policy.

Abstract

Single-view 3D visuomotor policies based on point-cloud observations, such as 3D Diffusion Policy, often suffer from limited viewpoint generalization due to partial observability and occlusion. This is particularly challenging for articulated objects whose internal motion induces geometry changes. To address this problem, we propose an articulation-driven digital twin augmentation approach that couples reconstruction, online articulation-state estimation, and observation generation within a shared 3D Gaussian Splatting (3DGS) model. Offline, we reconstruct an articulation-aware 3DGS part model and parameterize the articulation. Online, we optimize joint states through differentiable rendering conditioned on the base pose, then transform pre-sampled part point clouds using the estimated configuration. Fusing the object point cloud with robot geometry enables real-time augmentation from partial to global observations using single-camera input during execution. Experiments on the complete pipeline in simulation and the real world demonstrate accurate articulation tracking and improved viewpoint generalization of 3D Diffusion Policy.

Read PDF

Similar papers

Sep 2026

PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics

It is shown that a flexible and expressive transformer, PointZero, outperforms prior methods on the same data and demonstrates the utility of the pre-training objective by post-training PointZero for two downstream applications: action-conditioned 3D dynamics prediction and imitation learning.

B. P. Duisterhof, Kai-Feng Zhang, Adam Hung et al. · 6 citations · ⚡1
Sep 2026

VidAct: Learning Manipulation from In-the-Wild Videos with Object-Centric 3D Awareness

Video demonstrations offer a scalable alternative to costly robot data for learning manipulation, yet existing reconstruction-based approaches often rely on constrained camera viewpoints or human-to-robot retargeting, while the reconstructed trajectories are difficult to adapt to new objects configurations without dist...

Hang Li, Ming-Xin Zhang, Zi-Han Wu et al. · 1 citation
Preprint Oct 2026

3DROID: A Renderable 3D Gaussian Dataset with Measured Per-Scene Reliability

This work studies the effect of extrinsic reliability and pose conditioning on feed-forward 3DGS, and proposes a calibration-aware pipeline that anchors reconstructed scenes to the robot's metric workspace and presents a renderable, metric-pose-anchored dataset with scene-level reliability information for robot manipul...

W. Cho, Junhoo Lee, N. Kwak · 0 citations
Preprint Oct 2026

RoMo-GS: Efficient Online 3D Gaussian Splatting for Robot Mapping via Exposure-Integrated Motion Blur

Online 3D Gaussian Splatting (3DGS) is emerging as a practical way to build a representation for robotics applications. However, deploying it on real-world platforms exposes multiple challenges. Rapid maneuvers of a robot induce motion-blurred images, which corrupt the reconstructed geometry and cause camera pose drift...

Sang-Woo Jung, Dian Deng, Jia-Hui Fu et al. · 0 citations
#artificial intelligence Preprint Oct 2026

DepthWorld: 3D World Model for Robot Manipulation

This work introduces a calibration pipeline that combines learned stereo depth with a joint factor graph, pooling all episodes collected from the same physical robot to recover its shared kinematic parameters alongside per-scene extrinsics, and trains DepthWorld, a Stable Video Diffusion-based world model that jointly...

Jai Bardhan, Josef Sivic, Vladimír Petrík · 2 citations
Preprint Nov 2026

OfficeTwin: Realizing 3D Reconstruction of Redundant Scenes through Partial Viewpoint Perception

Feed-forward 3D reconstruction turns an uncalibrated monocular stream into a dense point cloud in a single inference pass, orders of magnitude faster than classical structure-from-motion (e.g., COLMAP), and reliable on the weakly textured, repetitive surfaces where keypoint matching fails outright. However, raw point c...

Quan-Yun Wu, Kyle Gao, Daisy Li et al. · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.