Skip to content

ReCAT: Remember, Count, and Time: Structured Recurrent Memory for Robot Manipulation

Sep 2026 · arXiv.org · Vol abs/2609.35200 · 0 citations · 42 references
Computer Science

TL;DR

Comparisons within ReCAT show that the observation encoder and every-block memory conditioning are needed for this performance, and that update rules developed for efficient sequence modeling behave differently as robot memory: additive updates have the highest observed success on counting and timing, and delta-rule updates on spatial recall.

Abstract

Memory-dependent manipulation requires robots to make decisions using information that is no longer available to their current sensors, such as recalling an earlier visual cue, tracking task progress, counting repeated events, or estimating elapsed time. We present ReCAT, a language-conditioned policy with structured recurrent memory. An instruction-conditioned encoder forms features from the current observation. A recurrent memory integrates the observation stream through Mamba-2 layers and one causal attention layer. A flow-matching Transformer decoder reads the current and the historical representation through separate cross-attention in every block. ReCAT reaches 95.3\% average success on LIBERO and 62.4\% on RMBench, with the best or tied-best result on six of nine tasks. On three real-robot tasks probing spatial recall, event counting, and interval timing, the best ReCAT variant reaches 66.7\% average success, against 8.3\% for the strongest short-history baseline. Controlled comparisons within ReCAT show that the observation encoder and every-block memory conditioning are needed for this performance. They also show that update rules developed for efficient sequence modeling behave differently as robot memory: additive updates have the highest observed success on counting and timing, and delta-rule updates on spatial recall. Project website is at https://intuitive-robots.github.io/ReCAT

View source

Similar papers

Simple Agentic Memory for Generalist Robot Policies

These results support a state-based view of robot memory: effective memory for control is not simply retained visual history, but compact task-relevant state derived from the interaction history.

Yu-You Zhang, Yun-Bei Zhang, Miao Li et al. · 1 citation

MemBodied: Recurrent Associative Memory for Vision-Language-Action Models

Findings support MemBodied as a practical alternative to expanding the policy context for history-dependent manipulation, a fixed-size episodic memory with two complementary components: an associative state that records interactions across policy calls and an episode anchor that preserves a compact representation of th...

Tej Deep Pala, N. Majumder, Bryce Goh et al. · 1 citation
#artificial intelligence Preprint Oct 2026

Divide-and-Remember: Recursive Action-Relevant Memory for Long-Horizon VLA Policies

Vision-language-action (VLA) models struggle on history-dependent manipulation tasks, where the current observation alone does not determine the action, and the policy needs a memory of the history. Existing memory methods decide what to remember by design, for example, keeping frames with large pixel changes, and show...

Xue-Hui Yu, Eason Yu, Mei-Yi Wang et al. · 0 citations
Preprint Sep 2026

T$^2$Mem: Learning Test-Time Memory for Robotics

Memory-dependent robotic manipulation requires policies to use information that is no longer available in the current observation. Retaining history alone is insufficient: memory must preserve information that supports future actions. One challenge is whether a memory-free foundation model can learn to retain and use h...

Yi-Ze Liu, Huang Huang, Yi-Ning Hong et al. · 2 citations
Preprint Oct 2026

Can a Frontier VLM Serve as a Robot Policy for Memory-Dependent Manipulation?

We use a frozen frontier vision-language model, GPT-6-Astra, directly as the robot policy on RoboMME, a benchmark of 16 memory-dependent manipulation tasks spanning temporal, spatial, object, and procedural memory. The model keeps the whole episode in context and outputs every end-effector waypoint and gripper command,...

Jack B. Jedlicki, Amish Sethi, Heng Yang · 1 citation · ⚡1

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.