Skip to content

The Memory Trap: Language Agents Overgeneralize from Benign Interactions

Nov 2026 · 1 citation · 79 references
Computer Science

TL;DR

It is shown that memory traps can arise both from overgeneralizations in memory writing, and when acting on memory, and thus motivate the need for new research into ensuring that agents preserve the scope of their memories.

Abstract

Language agents increasingly use persistent memory to record user preferences and procedures across sessions. We show that memory from benign interactions can cause unsafe or unreliable behavior on later tasks. We call this failure the memory trap. To study this issue, we compare behavior in sessions with and without memory across six frontier models (including Claude Sonnet, Opus, and Fable models and GPT-5.6). We evaluate these models in a range of settings. First, in autonomy experiments, we find that requests to skip confirmation during low-stakes coding chores lead agents to skip confirmation and verification on simulated high-stakes tasks in both coding and everyday life. We next study memories formed when users seek emotional support, or describe harm from uncritical feedback, and find these memories bias later judgments of mathematical work, arguments, and poems toward sycophancy or over-critique (respectively). Memory notes written during legitimate stale-test repairs lead agents to weaken correct tests that catch broken safeguards; these memories also increase cheating on ImpossibleBench, with larger increases for stronger Sonnet and Opus models. We show that memory traps can arise both from overgeneralizations in memory writing, and when acting on memory. These findings demonstrate that even benign interactions can lead to serious risks from memory systems, and thus motivate the need for new research into ensuring that agents preserve the scope of their memories.

View source

Similar papers

#human-computer interacti... Preprint Nov 2026

From Benchmarks to Behaviors: A Framework for Auditing AI Agents

Generative AI systems increasingly mediate access to information. Their behavior depends on tools, instructions, and user memories that are often difficult for researchers to observe or control. Here, I introduce TRACE (Transparent Research on Agent Contexts and Environments), a framework for systematically auditing ho...

Nick Hagar · 0 citations
#artificial intelligence Preprint Oct 2026

Understanding and Mitigating Inference-Time Overreliance Using Agentic Memory

MEMTRIM is proposed, a plug-and-play framework that indexes memory evidence at write time and controls its reuse at read time that removes repeated or conflicting evidence while preserving useful memory-specific information, requires no retraining, and applies to both embedding-based and structured memory systems.

Luo-Xi Tang, Yu-Qiao Meng, Nilesh Auradkar et al. · 0 citations
Preprint Nov 2026

Don't Just Trust Your Memory: Evidence-Linked Memory Reduces Memory-Induced Degradation in LLM Agents

External memory enables LLM agents to improve from experience by distilling trajectories of past tasks into concise lessons for new tasks. However, memory can also lead to memory-induced degradation, where an agent fails on tasks it would solve without memory, often because it follows a lesson that is incorrect, or mis...

Yukyeong Lee, Yumin Choi, Jinheon Baek et al. · 0 citations
#artificial intelligence Preprint Oct 2026

MemCut: Every Secret Leaves a Trail in the Long-Term Memory of LLM Agents

Long-term memory has become a crucial component of large language model (LLM) agents, with most research focusing on retrieving relevant information accurately and efficiently. However, a dependable memory system must also forget sensitive user data upon request. To evaluate this capability, we introduce MemCut, a syst...

Cheng Zhang, Xuan-Ning Cao, Zhi-Cheng Zheng et al. · 2 citations
Preprint Oct 2026

Looks Fine to Me: Understanding Oversight in Agentic Interactions

Agentic AI systems are deployed to execute complex tasks across domains such as writing, coding, and data analysis. As execution is offloaded to AI agents, the human role shifts toward oversight. Effective oversight requires recognizing when closer scrutiny is needed, even though flawed outputs may appear convincing. O...

Dinithi Dissanayake, Yasith Samaradivakara, Hyung Won Lee et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.