A unified memory-driven navigation system built around memory construction and management that transforms raw navigation history into compact, spatially grounded, selectively retrievable, and reusable navigation experience.
Abstract
Large vision-language models (LVLMs) have shown strong potential for embodied vision-language navigation (VLN), yet their performance degrades on long-horizon trajectories due to context-length limits, redundant historical observations, and unstable spatial alignment across time. These issues make it difficult to both compress long navigation histories and reliably reuse past spatial experience. To address these limitations, we propose \textbf{MemoryVLN}, a unified memory-driven navigation system built around memory construction and management. During memory construction, \emph{Long Sequence Compression} employs forgetting-curve-inspired temporal sampling to preserve dense recent evidence and sparse yet representative distant observations under a limited token budget. \emph{Ego-Aware Memory Representation} further integrates 2D semantic tokens, 3D geometric tokens, and 6D agent poses, allowing each memory to encode both what was observed and where it was observed. During memory management, \emph{Hierarchical Memory Bank} maintains in-episode and context-level memories through cross-context warm-start and confidence-controlled write-back. \emph{Memory Enhancement} then retrieves the Top-$K$ relevant memories according to the current navigation state and fuses them with the current visual tokens. Together, these mechanisms transform raw navigation history into compact, spatially grounded, selectively retrievable, and reusable navigation experience. Extensive experiments on R2R-CE and RxR-CE demonstrate consistent improvements over the state-of-the-art baselines by a substantial margin (i.e., 2\% and 6.5\% in successful rate).
Vision-language navigation (VLN) requires an embodied agent to follow language instructions through 3D environments. Maintaining persistent episodic memory remains a challenge for VLM-based streaming agents in long-horizon navigation. We present MEMVLN, a streaming episodic memory framework that maintains a bounded ban...
Yan Wang, Jiangyong Huang, Si-Yuan Huang et al.· 0 citations
Vision-Language-Action (VLA) models have shown remarkable success in robotic manipulation, yet their ability to execute long-horizon tasks remains limited by insufficient long- term memory modeling. Existing methods either rely on implicit temporal representations within policy networks, where historical information ca...
Xin-Yu Liu, Dai-Ying Tian, Chen-Wei Tang et al.· 0 citations
Vision-Language Navigation (VLN) requires an embodied agent to follow natural-language instructions and navigate in previously unseen environments. Despite recent progress in continuous multimodal navigation, long-horizon VLN remains challenging since it requires preserving useful history and maintaining decision consi...
Mu-He Ding, Xing-Yu Xie, Hao-Xiang Shi et al.· Proceedings of the 34th ACM...· 0 citations
Long-horizon robotic manipulation fundamentally requires memory: agents must retain task-relevant information from past observations and interactions and use it to guide future actions. Existing approaches typically preserve raw historical observations, compressed visual tokens, or language summaries, but these represe...
Yun Zhang, Zhi-Yu Huang, Tian-Hui Cai et al.· 1 citation
Bounded navigation-aware retention preserves navigation performance while considerably reducing GFM-KV memory relative to larger-memory temporal retention, which support jointly examining the geometric representations exposed to the policy and the historical evidence retained for future inference.
Q. Phạm, Anh Dao, Danh Vinh Le et al.· arXiv.org· 0 citations
Zero-shot vision-and-language navigation (VLN) requires an agent to use past observations without navigation-specific training. We study what information an 8B-scale navigation system should retain and how it should be organized for subsequent decisions. Our agent combines instruction progress, hierarchically compresse...
Yu-Hang Peng, Zhi-Kai Wang, Zi-Yi Wang et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduJul 30, 2026
From feet to fingertips — we are teaching robots intelligent whole-body control, fine dexterity, and teamwork to complete a broad range of complex tasks.
MIT News · Artificial Intelligence· news.mit.eduNov 18, 2026