Skip to content
Preprint

Latent-Foresight: End-to-End Learning Predictable Representations for Latent World Models

Oct 2026 · 0 citations · 58 references
Computer Science

TL;DR

Extensive experiments show that the proposed Latent-Foresight approach learns more temporally coherent latent representations and consistently outperforms two-stage baselines across multiple future scene understanding tasks and prediction horizons, while eliminating separate training stages, including during high-resolution adaptation.

Abstract

Predicting the future evolution of a scene is a fundamental capability for world modeling. Recent work has shown that operating in the feature space of Vision Foundation Models (VFMs) yields semantically rich representations that support diverse future scene understanding tasks. However, existing approaches rely on two-stage pipelines, where VFM features are first compressed using fixed dimensionality reduction (e.g., PCA) or independently trained autoencoders, and a separate predictor is trained on top of the resulting frozen latent space. This decoupling between representation learning and temporal prediction, as well as approaches that apply predictors directly on raw VFM features, provides no guarantee that the latent space is structured for predictable dynamics. In this work, we propose Latent-Foresight, an end-to-end framework that jointly learns a latent tokenizer and a flow-based generative dynamics model, explicitly shaping the representation to support temporal predictability. To enable stable joint optimization, we introduce several key design choices that prevent latent collapse and align reconstruction with generative objectives. Extensive experiments show that our approach learns more temporally coherent latent representations and consistently outperforms two-stage baselines across multiple future scene understanding tasks and prediction horizons, while eliminating separate training stages, including during high-resolution adaptation. We provide the implementation code and model weights at https://github.com/Sta8is/Latent-Foresight

View source

Similar papers

Preprint Sep 2026

WALT: Learning World-Model-Aligned Latent Trajectories for Autonomous Driving

The proposed World-Model Alignment for Latent Trajectories (WALT), which learns a compact generative trajectory latent space by transferring information from a frozen pretrained driving world model without modifying the world model itself, is proposed.

Ming-Kai Jia, Jia-Xin Guo, Zhi-Jian Shu et al. · 0 citations
Preprint Oct 2026

Juno: Taming Predictive Latents for Vision-Language-Action Models

Joint-embedding predictive architectures (JEPAs) predict masked or future observations in representation space, offering a natural source of predictive latents for vision-language-action (VLA) models. Yet making these latents useful across pretraining, policy learning, and deployment requires addressing three failures:...

Yu-Chen Zhu, Chen-Yi Xu, Yu-Lin Zhang et al. · 0 citations
Open access Sep 2026

Latent Space Representation Learning Based on Variational Autoencoders

The study demonstrates that the VAE possesses significant advantages in state compression and uncertainty modeling, but suffers from issues such as blurry image generation and oversimplified posterior distribution assumptions.

Xiao-Zhou Gao · 0 citations
Preprint Aug 2026

LeFlow: Generative Latent Flow Planning for World Models

This work presents LeFlow, which learns a reusable latent trajectory prior operating directly in the latent dynamics space from the world model, and argues that latent world models should support not only prediction but reusable planning priors.

Hsiang-Wei Huang, Jian-Xu Shangguan, Junbin Lu et al. · 5 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.