Skip to content
Preprint

Future Representation Distillation for Imagination-free World Action Models

Nov 2026 · 0 citations · 35 references
Computer Science

TL;DR

Experiments show that DreamFree can translate stronger video priors and additional video data into better robot control and enable learning new robot tasks from video demonstrations, including videos from a different embodiment, without action annotation for the new tasks.

Abstract

World Action Models (WAMs) use the predictive knowledge of pretrained video diffusion models for robot control through future video prediction. However, iterative video denoising is costly, and inaccurate visual predictions can compromise action prediction. Our key idea is to make this predictive knowledge available through visual representations computed from the current observation, enabling action prediction without generating future videos. We introduce DreamFree, a WAM that repurposes a pretrained video diffusion model as a predictive visual encoder through representation distillation. The visual encoder learns to predict a frozen video diffusion model's representation using only the current observation. An action module is then trained to predict actions by extracting action-relevant information from the visual encoder. Across real-world and simulated tasks, DreamFree achieves higher success rates than state-of-the-art VLAs and WAMs, while performing real-time closed-loop control at 11Hz with a 14B video diffusion backbone. Experiments show that DreamFree can translate stronger video priors and additional video data into better robot control and enable learning new robot tasks from video demonstrations, including videos from a different embodiment, without action annotation for the new tasks.

View source

Similar papers

Preprint Oct 2026

Self-Distilling Future Video Representations for World Action Models

Future REpresentation learning via self-Distillation with temporal context (FRED), a framework that trains WAMs to improve the quality of future video representations from noise in a single denoising step, and substantially improves human-to-robot skill transfer.

Huiwon Jang, Suhyeok Jang, Hyeon-Joo Kang et al. · 0 citations

Sparse-WAM: Accelerating World Action Models via Action-Guided Sparse Imagination

Pilot, an efficient engine that reduces sparse inference overhead through lightweight scoring and cross-step reuse of token selections, is introduced, an efficient engine that reduces sparse inference overhead through lightweight scoring and cross-step reuse of token selections.

Xin-Ling Xie, Hao-Dong Wang, Jia-Zhi Mi et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Adapting Video Models for Robot Control via Motion-guided Asymmetric Prediction

Motion-Guided Asymmetric Prediction (MoGAP), which augments WAM training with motion-focused visual prediction, which consistently improves policy performance and uses motion cues derived directly from training videos, requiring neither an external teacher nor additional modalities.

Taeyoung Kim, Junyoung Sung, Heeseung Kwon et al. · 0 citations
Sep 2026

The Right Future for Action: Learning Action-Relevant Predictive States in World Action Models

These findings motivate Action-Relevant Predictive States (ARPS), a compact predictive interface between the video and action experts that achieves 99.2% success on LIBERO and transfers to LIBERO-Plus without adaptation, reaching 87.3% and exceeding Fast-WAM by 39.2 percentage points.

Qi-Wen Gu, Ji-Fan Li, Bing-Jie Gao et al. · 8 citations · ⚡1
Preprint Oct 2026

IronMan: Information-Constrained Video-Action Learning for Robot Manipulation

Video Action Models (VAMs) couple visual dynamics modeling with action generation for robot manipulation. However, video representations are not naturally suited to action generation, as exposing the action policy to excessive visual detail can impair its generalization ability. Therefore, we introduce IronMan (Informa...

Yuan-Shuo Zhang, Wen-Zhe Zhao, Zi-Xing Lei et al. · 0 citations
Preprint Oct 2026

Video Prediction Policy 2: Predict Better, Act Better

World action models (WAMs) have emerged as an important class of generalist robot policies, aiming to transfer video prediction priors to action learning. However, we find that existing WAMs frequently produce incorrect motion predictions in open-ended environment, leading to erroneous actions. We attribute this limita...

Yanjiang Guo, Hao-Dong Yan, Zhi-De Zhong et al. · 9 citations · ⚡1

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.