Skip to content

Adapting Video Models for Robot Control via Motion-guided Asymmetric Prediction

Oct 2026 · 0 citations · 39 references
Computer Science

TL;DR

Motion-Guided Asymmetric Prediction (MoGAP), which augments WAM training with motion-focused visual prediction, which consistently improves policy performance and uses motion cues derived directly from training videos, requiring neither an external teacher nor additional modalities.

Abstract

World Action Models (WAMs) leverage pretrained video generative models for robotic control by jointly predicting future visual observations and actions. The visual prediction objective plays a key role in adapting pretrained video representations to the target robot domain. Yet standard visual prediction inherited from video pretraining treats all visual regions uniformly, despite their differing relevance to robotic control. We therefore investigate the spatial allocation of visual supervision in WAMs and show that focusing supervision on dynamic regions can substantially improve policy performance. Building on this finding, we introduce Motion-Guided Asymmetric Prediction (MoGAP), which augments WAM training with motion-focused visual prediction. MoGAP allocates noise asymmetrically across regions, training the model to reconstruct more heavily corrupted dynamic regions using less corrupted static regions as context. MoGAP is trained alongside standard video-action prediction through a training-only auxiliary prediction path within the shared video diffusion backbone. It uses motion cues derived directly from training videos, requiring neither an external teacher nor additional modalities. Across robot manipulation benchmarks and WAM architectures, MoGAP consistently improves policy performance, increasing success rates on the GR-1 tabletop benchmark from 38.8% to 47.5% for Fast-WAM (Wan2.2-5B) and from 52.3% to 62.3% for Cosmos3-Nano.

View source

Similar papers

Preprint Oct 2026

Video Prediction Policy 2: Predict Better, Act Better

World action models (WAMs) have emerged as an important class of generalist robot policies, aiming to transfer video prediction priors to action learning. However, we find that existing WAMs frequently produce incorrect motion predictions in open-ended environment, leading to erroneous actions. We attribute this limita...

Yanjiang Guo, Hao-Dong Yan, Zhi-De Zhong et al. · 9 citations · ⚡1
Preprint Oct 2026

Trace-WAM: Learning to Act via 3D Motion Prediction for World Action Models

World Action Models (WAMs) learn representations for robot control by coupling future visual prediction with action generation. Successful manipulation depends primarily on how the robot and task-relevant objects evolve, whereas future-video prediction must also reconstruct the appearance and background of the entire s...

Xu-Qing Hu, Chang Jie Wu, Lin-Zhe Shi et al. · 0 citations
Preprint Nov 2026

Future Representation Distillation for Imagination-free World Action Models

Experiments show that DreamFree can translate stronger video priors and additional video data into better robot control and enable learning new robot tasks from video demonstrations, including videos from a different embodiment, without action annotation for the new tasks.

Jaehyeong Jo, Sang-Sub Jang, Sihyun Yu et al. · 0 citations
#artificial intelligence Preprint Oct 2026

World Action Modeling with Progressive Visual Planning

World action models (WAMs) have emerged as a promising paradigm for robotic control by jointly predicting future visual dynamics and actions from an initial observation and instruction. However, existing WAMs struggle with long-horizon prediction, as generating dense video rollouts is highly inefficient. Some recent WA...

Fei Zhang, Zhao-Chong An, D. Frost et al. · 1 citation
Preprint Oct 2026

Native Action-Prior Learning from Videos for World Action Models

World action models integrate future visual dynamics with robot action prediction, but their scalability remains limited by the need for action-annotated robot trajectories. Observation-only videos contain rich evidence about interaction dynamics, but existing approaches typically use them either to pretrain visual rep...

Zhao-Chong An, Fei Zhang, Meng-Lin Jia et al. · 0 citations

Copper-Policy: Focus on the Representation for Robust Robot Manipulation

Copper-Policy is introduced, which learns a compact World representation with the policy rather than relying on a predefined target space and predicts future observation embeddings conditioned on task intention without reconstructing pixels through temporal joint-embedding prediction.

Ze Feng, Yi-Xu Feng, Ling-Yu Xiao et al. · 1 citation

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.