Experiments show that DreamFree can translate stronger video priors and additional video data into better robot control and enable learning new robot tasks from video demonstrations, including videos from a different embodiment, without action annotation for the new tasks.
Abstract
World Action Models (WAMs) use the predictive knowledge of pretrained video diffusion models for robot control through future video prediction. However, iterative video denoising is costly, and inaccurate visual predictions can compromise action prediction. Our key idea is to make this predictive knowledge available through visual representations computed from the current observation, enabling action prediction without generating future videos. We introduce DreamFree, a WAM that repurposes a pretrained video diffusion model as a predictive visual encoder through representation distillation. The visual encoder learns to predict a frozen video diffusion model's representation using only the current observation. An action module is then trained to predict actions by extracting action-relevant information from the visual encoder. Across real-world and simulated tasks, DreamFree achieves higher success rates than state-of-the-art VLAs and WAMs, while performing real-time closed-loop control at 11Hz with a 14B video diffusion backbone. Experiments show that DreamFree can translate stronger video priors and additional video data into better robot control and enable learning new robot tasks from video demonstrations, including videos from a different embodiment, without action annotation for the new tasks.
Future REpresentation learning via self-Distillation with temporal context (FRED), a framework that trains WAMs to improve the quality of future video representations from noise in a single denoising step, and substantially improves human-to-robot skill transfer.
Huiwon Jang, Suhyeok Jang, Hyeon-Joo Kang et al.· 0 citations
Pilot, an efficient engine that reduces sparse inference overhead through lightweight scoring and cross-step reuse of token selections, is introduced, an efficient engine that reduces sparse inference overhead through lightweight scoring and cross-step reuse of token selections.
Xin-Ling Xie, Hao-Dong Wang, Jia-Zhi Mi et al.· arXiv.org· 0 citations
Motion-Guided Asymmetric Prediction (MoGAP), which augments WAM training with motion-focused visual prediction, which consistently improves policy performance and uses motion cues derived directly from training videos, requiring neither an external teacher nor additional modalities.
Taeyoung Kim, Junyoung Sung, Heeseung Kwon et al.· 0 citations
These findings motivate Action-Relevant Predictive States (ARPS), a compact predictive interface between the video and action experts that achieves 99.2% success on LIBERO and transfers to LIBERO-Plus without adaptation, reaching 87.3% and exceeding Fast-WAM by 39.2 percentage points.
Video Action Models (VAMs) couple visual dynamics modeling with action generation for robot manipulation. However, video representations are not naturally suited to action generation, as exposing the action policy to excessive visual detail can impair its generalization ability. Therefore, we introduce IronMan (Informa...
Yuan-Shuo Zhang, Wen-Zhe Zhao, Zi-Xing Lei et al.· 0 citations
World action models (WAMs) have emerged as an important class of generalist robot policies, aiming to transfer video prediction priors to action learning. However, we find that existing WAMs frequently produce incorrect motion predictions in open-ended environment, leading to erroneous actions. We attribute this limita...