Skip to content

WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining

Oct 2026 · 0 citations · 55 references
Computer Science

TL;DR

WorldFlow achieves the lowest average displacement error (ADE) and final displacement error (FDE) among the evaluated learning-based methods, and is introduced as a generative world model that jointly predicts future video and query-based 3D trajectories.

Abstract

Predicting future physical states stands as a central challenge in embodied intelligence. Existing video world models leave 3D dynamics implicit, action-based formulations remain tied to specific robot configurations, and 3D motion prediction models use flow-only supervision, providing a limited learning signal for physical dynamics. To circumvent these limitations, we introduce \method{}, a generative world model that jointly predicts future video and query-based 3D trajectories. By parameterizing 3D motion as continuous displacement tokens aligned with the video grid, a unified Mixture-of-Transformers (MoT) backbone jointly denoises both modalities under a rectified-flow objective. This formulation couples explicit geometric dynamics with visual evolution, effectively unlocking the physical commonsense embedded within pretrained video models. To empirically scale and validate this unified architecture, we train \method{} on a heterogeneous mixture of 1.44 million object-level samples spanning simulated and real-world domains. We further introduce WorldFlowBench, comprising 1,678 evaluation samples from 11 data sources across simulated and real-world interactions, including spatial reasoning tasks. Across all three evaluation sets, WorldFlow achieves the lowest average displacement error (ADE) and final displacement error (FDE) among the evaluated learning-based methods. Notably, on the challenging SpatialCog set, WorldFlow achieves an ADE of 0.060 m, a 54.2\% reduction compared with the strongest external baseline. It also reduces ADE and FDE by 55.9\% and 68.5\%, respectively, compared with its flow-only variant. Project page: https://worldflow-rbs.github.io/.

View source

Similar papers

Preprint Oct 2026

Trace-WAM: Learning to Act via 3D Motion Prediction for World Action Models

World Action Models (WAMs) learn representations for robot control by coupling future visual prediction with action generation. Successful manipulation depends primarily on how the robot and task-relevant objects evolve, whereas future-video prediction must also reconstruct the appearance and background of the entire s...

Xu-Qing Hu, Chang Jie Wu, Lin-Zhe Shi et al. · 0 citations
Sep 2026

PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics

It is shown that a flexible and expressive transformer, PointZero, outperforms prior methods on the same data and demonstrates the utility of the pre-training objective by post-training PointZero for two downstream applications: action-conditioned 3D dynamics prediction and imitation learning.

B. P. Duisterhof, Kai-Feng Zhang, Adam Hung et al. · 6 citations · ⚡1
Sep 2026

StrucPhysVideo: Learning Physical Dynamics from Structured Captions and Robot Actions

StrucPhysVideo, a family of video world models that bridges physics-focused data curation with language- and action-conditioned prediction of scene evolution, is presented, and a sparse Mixture-of-Experts text-image-to-video model trained with a curriculum that progressively emphasizes physical dynamics while retaining...

En-Hui Ma, Kai-Wen Guo, Ting-Rui Zhang et al. · 0 citations
Preprint Oct 2026

E0.5: From Reconstructive to Future-Predictive 3D Foundations for Robotic World-Action Modeling

World-action models (WAMs) offer a promising framework for embodied intelligence, yet mainstream approaches remain centered on video foundation models or vision-language models. Even when geometric features extracted by 3D models such as VGGT are incorporated, they are typically passed into a video model or VLM that se...

Xiao Lai, Zhi-Hao Zhan, Zhi-Quan Liao et al. · 1 citation
#machine learning Preprint Nov 2026

PlayGen2: Learning Real-Time Playable Worlds with Explicit 3D State

PlayGen2, a real-time interactive world model that separates learned 3D state evolution from neural rendering, is presented and zero-shot deployment on unseen Doom maps, multiplayer interaction through the composition of single-player models, and programmable gameplay through rule-based state interventions are demonstr...

Ji-Hai Zhang, Yu-Han Zhou, Jun-You Li et al. · 0 citations
Sep 2026

MVG-WAM: Multiple View Geometry-Aware World-Action Modeling for Robotic Manipulation

World-Action Models (WAMs) couple visual dynamics with action prediction, bringing the rich priors of pretrained video models to robotic manipulation. However, their multi-view interfaces typically tile images or concatenate tokens, leaving the geometric relationships among synchronized cameras implicit. This makes it...

Wen-Bo Chen, Tian-Fu Li, Hao-Xuan Xu et al. · 1 citation

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.