Learning what to Remember: Reinforcement Learning for Cost-Aware Memory in LLM Agents

Abstract

Long-term memory is a critical capability for LLM-based agents, yet most current systems rely on RetrievalAugmented Generation (RAG), which stores information indiscriminately and retrieves by similarity alone. This passive approach lacks any notion of utility: it does not distinguish valuable knowledge from noise and incurs unbounded storage costs. This constraint is particularly critical in human-robot interaction (HRI), where social robots and personal assistants need to engage with users over extended periods and need to retain concise, efficiently searchable records of user preferences and past interactions. We propose MemRL, a framework that treats memory as a learned, cost-aware policy rather than a static database. Using a study-exam paradigm with synthetic tasks inaccessible to pretrained models, we train a lightweight write policy with REINFORCE to decide which pieces of information to store in memory. Across multiple random seeds and storage penalty levels, the learned policy achieves the same exam accuracy as a store-all RAG baseline while using 80% less memory, resulting in a substantially higher cost-aware performance. These findings demonstrate that selective, utility-driven memory is both practical and superior to passive logging, paving the way for cognitively-inspired memory systems in autonomous agents.