Skip to content
Preprint

World2Motion: Turning Video World Models into 3D Human Motion Generators

Sep 2026 · 0 citations · 51 references
Computer Science

TL;DR

World2Motion, a framework that generates scene-aware 3D human motion and corresponding video from a single image and a text prompt, and proposes a shift-decoupled noise schedule that assigns different noise levels to video and motion through shared denoising progress, reducing motion jitter.

Abstract

We present World2Motion, a framework that generates scene-aware 3D human motion and corresponding video from a single image and a text prompt. While existing 3D motion generators learn from motion datasets, their generalization is constrained by limited coverage of environments. In contrast, video world models such as Cosmos 3 offer broader environmental priors but are not designed for full-body motion generation; recovering motion from their generated videos requires costly two-stage inference. To address these, we turn Cosmos 3 into a single-stage 3D motion generator. This adaptation has two challenges: the scarcity of paired video--motion data and temporal instability in the generated motion. First, we construct a training dataset combining synthetic video--motion pairs with real videos paired with estimated 3D motion. Second, we propose a shift-decoupled noise schedule that assigns different noise levels to video and motion through shared denoising progress. This design accommodates the different denoising requirements of the two modalities, reducing motion jitter. Experiments on a multi-source interaction benchmark show that World2Motion has better motion--text alignment and scene interaction compared with the evaluated 3D motion generators. It also matches the interaction success rate of the two-stage baseline while achieving approximately 3.3$\times$ faster inference. Our project page is available at https://fyantu.github.io/World2Motion/.

View source

Similar papers

Preprint Oct 2026

FlowHMR: Physically Plausible Motion Capture from Video

We present FlowHMR, a framework for recovering physically plausible global 3D human motion from monocular video. Previous learning-based methods typically regress human motion directly from video and train the network with geometric supervision. However, recovering human motion from monocular video is inherently ambigu...

Zhan-Ke Wang, Cheng-Feng Zhao, Qing Shuai et al. · 0 citations
#artificial intelligence Preprint Oct 2026

DreamHMR: From Video Generation to Human Motion Modeling via Structural Focusing

Human mesh recovery (HMR) provides a geometric foundation for understanding human motion, but accurately reconstructing both body motion and fine hand articulation from videos remains challenging, especially in real-world scenes where occlusion and partial visibility are common. Video generation pretraining offers rich...

Zheng-Jie Xu, Rui-Hao Gong, Xiang-Long Liu · 0 citations
Preprint Oct 2026

Parasitic Co-Denoising: Unlocking 3D Human Motion Generation in a Frozen Video Diffusion Model

Despite never being supervised on explicit 3D motion, large-scale text-to-video diffusion models synthesize realistic human motion in their generated videos. We ask whether this implicit knowledge can be turned into explicit 3D motion generation, without training a separate motion model. Probing a frozen Wan2.1 reveals...

Yun-Jiao Zhou, Junlang Qian, Li-Hua Xie et al. · 0 citations
Preprint Sep 2026

Kirin: Animal Motion Generation from In-the-Wild Video

Kirin is introduced, a framework that reconstructs motion from video, learns motion priors at scale, and generates realistic motion that can be directly applied to animated assets that establishes a new foundation for large-scale, text and image conditioned animal motion generation and animation.

Brian Nlong Zhao, Zhuo-Yang Pan, J. Rehg et al. · 1 citation
Preprint Sep 2026

Seeing the World and the Self from Egocentric Video

Experiments show that RESELF outperforms state-of-the-art methods designed for the individual tasks across depth estimation, camera tracking, and full-body motion estimation.

Kai Guan, Minchao Jiang, Ruichen WangLi et al. · 0 citations
Preprint Oct 2026

Image Diffusion Models Are Good 3D Human Motion Generators

Text-to-motion generation typically relies on dataset-specific skeletal representations, with models trained separately for benchmarks with different joint layouts. We investigate whether image diffusion architectures can model 3D human motion through a common image-compatible representation that accommodates heterogen...

Si-Yuan Yang, Chong Wang, Hao Cheng et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.