2025· Neural Information Processing Systems· pp. 110988-111013· 3 citations· 78 references
Computer Science
TL;DR
This work introduces VideoTitans, the first architecture to adapt the gradient-driven Titans memory—originally designed for language modelling to video prediction, and achieves competitive visual fidelity compared to state-of-the-art recurrent, convolutional, and efficient-transformer methods.
Abstract
Accurate video forecasting enables autonomous vehicles to anticipate hazards, robotics and surveillance systems to predict human intent, and environmental models to issue timely warnings for extreme weather events. However, existing methods remain limited: transformers rely on global attention with quadratic complexity, making them impractical for high-resolution, long-horizon video prediction, while convolutional and recurrent networks suffer from short-range receptive fields and vanishing gradients, losing key information over extended sequences. To overcome these challenges, we introduce VideoTitans , the first architecture to adapt the gradient-driven Titans memory—originally designed for language modelling to video prediction. VideoTitans integrates three core ideas: (i) a sliding-window attention core that scales linearly with sequence length and spatial resolution, (ii) an episodic memory that dynamically retains only informative tokens based on a gradient-based surprise signal, and (iii) a small set of persistent tokens encoding task-specific priors that stabilize training and enhance generalization. Extensive experiments on Moving-MNIST, Human3.6M, TrafficBJ and WeatherBench benchmarks show that VideoTitans consistently reduces computation (FLOPs) and achieves competitive visual fidelity compared to state-of-the-art recurrent, convolutional, and efficient-transformer methods. Comprehensive ablations confirm that each proposed component contributes significantly.
LongVU-TTT is introduced, which inserts a convolutional Test-Time Training (TTT) resampler with causal fast-weight updates between the vision encoder and the LLM, and is stronger than attention- and fixed-state recurrent resamplers across three benchmarks.
Mahmoud Ahmed, Sameh Abdulah, Olatunji Ruwase et al.· 0 citations
This paper introduces TinyMem, a model built upon compact multimodal memory for long-form video action detection that outperforms a range of state-of-the-art models on AVA v2.2 while using 5 times fewer memory tokens than the baseline with dense visual memory embeddings.
Rui Tian, Qi Dai, Hang-Rui Hu et al.· IEEE Transactions on Pattern...· 0 citations
AlayaWorld is presented, an interactive long-horizon video world model that generates 24-fps video at 540p and 720p and introduces a discrete autoregressive distillation formulation that combines distribution-matching distillation, self-forcing++, and consistency distillation, reducing inference from approximately 30 sampling steps to four steps per chunk.
AlayaWorld Team Kaipeng Zhang, Chuanhao Li, Y. Zhan et al.· arXiv.org· 4 citations
Long-horizon future-frame prediction is important for autonomous driving, traffic surveillance, and intelligent transportation systems, yet remains challenging due to temporal ghosting, geometry drift, and inconsistent object motion. Recent latent video diffusion models have achieved impressive visual quality, but directly applying them to structured traffic scenes often leads to unstable geometry and degraded temporal coherence over extended horizons. We present a training-free inference framework that stabilizes reliable static structure in pretrained video predictions through multi-frame temporal context and view-conditioned routing. For front-camera videos, our method refines generated futures with a multi-frame depth-layered renderer that projects static geometry from observed history frames while preserving dynamic regions from the generative base model. For heterogeneous traffic views, a frozen vision-language model infers a coarse camera group from the observed clip and selects a specialized motion-based predictor. The framework requires neither retraining nor fine-tuning of the underlying video model and can be applied directly to pretrained generators. We validate the proposed framework on the AI City Challenge Track 5 benchmark, where our final system achieves competitive performance among the top-ranked teams. These results demonstrate that geometry-aware inference-time refinement and view-conditioned hybrid inference can improve static-geometry stability and low-level structural fidelity without changing the original model architecture.
K. M. Le, H. Pham, Luu Thanh Danh et al.· 1 citation
MiniWorld is presented, a reproducible framework for training streaming video world models from scratch that adopts a chunk-wise non-decreasing noise schedule and two-stage continued training to improve temporal modeling and stability and will facilitate future research on video world modeling.
Yian Zhao, Ruochong Zheng, Hongcan Guo et al.· 0 citations
HeadCast is proposed, a training-free, plug-and-play acceleration framework built on the observation that a pre-trained AR model's attention heads exhibit stable, heterogeneous behaviors that accelerates inference by up to 1.62x at 720P and 1.95x at 1080P, while keeping VBench quality on par with full attention and largely flicker-free.
Jinliang Shen, Li Su, Zheming Li et al.· arXiv.org· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
