Skip to content
Preprint

MEMO: Multi-Level Entity-Aware Memory for Streaming Video Understanding

Sep 2026 · 1 citation · 50 references
Computer Science

TL;DR

This work argues that streaming video memory should explicitly encode structured and semantically meaningful representations, particularly at the entity level, and proposes MEMO, a novel framework that models streaming video through multi-level, entity-aware structured memory.

Abstract

Streaming video understanding requires models to process unbounded visual streams while preserving rich visual semantics across vast temporal horizons, posing a fundamental challenge for memory modeling. Existing approaches primarily focus on increasing memory capacity, either by compressing historical information into fixed-size representations or by extending storage beyond GPU memory. However, these methods largely rely on global or coarse-grained representations, inevitably losing fine-grained visual information. In this work, we argue that streaming video memory should explicitly encode structured and semantically meaningful representations, particularly at the entity level. To this end, we propose MEMO, a novel framework that models streaming video through multi-level, entity-aware structured memory. MEMO performs multi-level perception to jointly capture global semantics, entity dynamics, and spatial structures, partitioning streaming video into semantically coherent chunks. Each chunk is organized into a structured memory, where lightweight global and entity-level representations serve as retrieval indices, while the corresponding high-resolution visual content is retained separately for on-demand access. At inference time, MEMO performs query-specific retrieval over the structured memory and selectively recalls relevant visual evidence for downstream reasoning. Notably, MEMO is training-free and plug-and-play with existing multimodal large language models. Extensive experiments on StreamingBench and OVO-Bench demonstrate that MEMO consistently improves multiple base models and achieves state-of-the-art performance. The code is available at https://github.com/Cecilia384/MEMO .

View source

Similar papers

#artificial intelligence Preprint Sep 2026

MemEvo: Automatic Discovery of Streaming Video Memory Mechanisms

Query-agnostic streaming video understanding requires vision-language models to continuously compress an indefinitely growing visual stream into a bounded memory before future queries are known. The performance depends critically on the memory mechanism--what observations to preserve, how to represent and consolidate t...

Guohong Liu, Jia-Lei Ye, Shan-Hui Zhao et al. · 0 citations
Preprint Aug 2026

StreamEMS: Streaming Video Understanding with Self-Evolving Memory Scheme for Vision-Language Models

Recently, many streaming video understanding methods have been proposed by constructing an external memory to store historical data for computational reduction. Most methods focus on optimizing the injection procedure of current data (write) and retrieving informative historical data (read) from memory, while overlooki...

Yu-Xing Liu, Pei-Qin Zhuang, Ya-Li Wang · 1 citation
Preprint Aug 2026

Dynamic Hub-and-Spoke Memory for Streaming Video Understanding

Dynamic Hub-and-Spoke Memory is proposed, a training-free framework that represents distant history as structured textual memory while preserving the recent frames as visual tokens for fine-grained perception in streaming video understanding.

Xin-Ru Jiang, Lin Zhao, Xi Xiao et al. · 4 citations
#artificial intelligence Preprint Oct 2026

ReMem: Streaming Video Understanding With Long Context Retention

Despite their impressive performance on a wide range of video understanding tasks, current Vision Language Models (VLMs) are predominantly designed for offline scenarios and struggle to handle online streaming videos that demand low latency response. Several studies have explored memory and token compression strategies...

Yi-Heng Li, Xu He, Shao-Bo Wang et al. · 0 citations
Preprint Sep 2026

Beyond Retrieval: Progressive Latent Memory Evolution for Streaming Video Understanding

This work introduces LatentStream, a progressive latent working memory framework that shifts streaming memory from store-and-retrieve to retrieve-and-internalize, and achieves new state-of-the-art results on existing online and offline video benchmarks.

Hongyu Qu, Guang-Ming Yao, Ling Xing et al. · 0 citations
#artificial intelligence Preprint Sep 2026

KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding

KeyRec is a training-free framework for constructing bounded visual memory that preserves fine-grained recent observations in a visual cache and organizes historical evidence into a structured event bank and achieves the best compressed performance in 13 of 15 settings using only 10\% of the dense decoder-facing visual...

Zi-Han Chen, Xuejian Rong, Xiao-Juan Wang et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.