Extensive experiments show that the proposed Latent-Foresight approach learns more temporally coherent latent representations and consistently outperforms two-stage baselines across multiple future scene understanding tasks and prediction horizons, while eliminating separate training stages, including during high-resolution adaptation.
Abstract
Predicting the future evolution of a scene is a fundamental capability for world modeling. Recent work has shown that operating in the feature space of Vision Foundation Models (VFMs) yields semantically rich representations that support diverse future scene understanding tasks. However, existing approaches rely on two-stage pipelines, where VFM features are first compressed using fixed dimensionality reduction (e.g., PCA) or independently trained autoencoders, and a separate predictor is trained on top of the resulting frozen latent space. This decoupling between representation learning and temporal prediction, as well as approaches that apply predictors directly on raw VFM features, provides no guarantee that the latent space is structured for predictable dynamics. In this work, we propose Latent-Foresight, an end-to-end framework that jointly learns a latent tokenizer and a flow-based generative dynamics model, explicitly shaping the representation to support temporal predictability. To enable stable joint optimization, we introduce several key design choices that prevent latent collapse and align reconstruction with generative objectives. Extensive experiments show that our approach learns more temporally coherent latent representations and consistently outperforms two-stage baselines across multiple future scene understanding tasks and prediction horizons, while eliminating separate training stages, including during high-resolution adaptation. We provide the implementation code and model weights at https://github.com/Sta8is/Latent-Foresight
The proposed World-Model Alignment for Latent Trajectories (WALT), which learns a compact generative trajectory latent space by transferring information from a frozen pretrained driving world model without modifying the world model itself, is proposed.
Ming-Kai Jia, Jia-Xin Guo, Zhi-Jian Shu et al.· 0 citations
Joint-embedding predictive architectures (JEPAs) predict masked or future observations in representation space, offering a natural source of predictive latents for vision-language-action (VLA) models. Yet making these latents useful across pretraining, policy learning, and deployment requires addressing three failures:...
Yu-Chen Zhu, Chen-Yi Xu, Yu-Lin Zhang et al.· 0 citations
Findings support the use of the Fourier auxiliary head method to improve both overall success rate and data efficiency, while avoiding representation laziness in latent world models.
Peng Zhu, Salvatore Penachio, Kaustav Mukherjee et al.· 0 citations
The study demonstrates that the VAE possesses significant advantages in state compression and uncertainty modeling, but suffers from issues such as blurry image generation and oversimplified posterior distribution assumptions.
Xiao-Zhou Gao· Theoretical and Natural Scie...· 0 citations
This work presents LeFlow, which learns a reusable latent trajectory prior operating directly in the latent dynamics space from the world model, and argues that latent world models should support not only prediction but reusable planning priors.
Hsiang-Wei Huang, Jian-Xu Shangguan, Junbin Lu et al.· 5 citations
P predictive visual latents are established as a foundation for effective WAM learning from task-specific demonstrations and for transferring future-modeling knowledge acquired from broader in-the-wild videos.
Yang Zhang, Jiang-Yuan Zhao, Chen-You Fan et al.· arXiv.org· 3 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.