ReMEMBER is proposed, a missing-evidence memory framework that conditions retrieval on unresolved window dependencies and refines retrieved chunks into evidence-dense memory under a fixed budget and improves memory recall and gap-resolution completeness over memory construction baselines under the same budget.
Abstract
Users of modern platforms repeatedly need summaries of recent dialogue, but the window rarely contains enough context to be interpreted on its own. We formalize this setting as streaming dialogue summarization, where a system must summarize a current window using selective memory from an unbounded history under a fixed budget. We show that the central challenge is not how much history is accessed, but whether memory recovers the evidence that the current window presupposes. We construct a benchmark and evaluation protocol that separately assesses whether memory contains gap-resolving evidence and whether the generated summary reflects it. We propose ReMEMBER, a missing-evidence memory framework that conditions retrieval on unresolved window dependencies and refines retrieved chunks into evidence-dense memory under a fixed budget. Experiments on dialogues with histories up to 160K tokens show that ReMEMBER improves memory recall and gap-resolution completeness over memory construction baselines under the same budget.
Long-term conversational agents rely on personal memory to maintain coherence and personalization, yet practical systems must operate under context budgets and cope with evolving or contradictory user information. We frame persona memory as a retrieval problem over a growing memory store, and propose REMAP, a reflection-guided memory editing approach for online alignment of persona facts that selectively writes and revises memory entries based on the current dialogue evidence and retrieved related items. The method aims to preserve salient facts while reducing redundancy and resolving apparent conflicts, enabling more efficient context utilization over extended interaction horizons. Experiments on multi-session dialogue datasets show consistent gains in persona-consistent retrieval and response continuity over commonly used memory strategies, while achieving more selective memory updates under comparable operational overhead.
Qingyang Xu, Xiao Liu, Zhou Fang et al.· Annual International ACM SIG...· 0 citations
Large language models (LLMs) are increas- ingly deployed as long-horizon conversational agents, motivating growing interest in mem- ory systems. However, existing benchmarks primarily evaluate memory through QA-style probing rather than in-situ conversational usage. We introduce LOCOMO-CONV, a conversa- tional memory benchmark derived from Lo- CoMo with four query styles: dialog, implicit, counterfactual, and composed. Across five rep- resentative memory systems, we evaluate both retrieval recall and end-to-end response qual- ity. Our experiments show that conversational framing exposes substantial retrieval gaps over- looked by QA benchmarks, especially on im- plicit and composed queries, which multi-facet query rewriting narrows for raw-turn mem- ory but not abstractive memory. We further find that strong retrieval does not fully trans- late into response quality, and that implicit queries exhibit silent grounding, where mem- ory improves contextual grounding without ex- plicitly surfacing the gold fact. These results point to reasoning-based memory elaboration as a promising direction, and we release aux- iliary supportive_memory annotations captur- ing conversationally useful context beyond the original gold evidence.
Long-term conversational agents rely on memory databases to ensure consistent user state across multiple interactions. Although prior work has analyzed retrieval-augmented generation and persistent memory for conversational agents, few studies have evaluated how memory-write policies influence memory retrieval behavior later on. We investigate whether memory interference originates mainly from memory retrieval or from the accumulation of competing fact versions added during memory updates. Three memory-write policies were evaluated in a controlled virtual patient dialogue environment. Append-only retrieval (NAIVE-SLOT), append-only retrieval with recency ranking (RECENT-SLOT), and slot-overwrite memory (SMART-SLOT) which maintains a single canonical value for each fact. Three clinical scenarios were implemented, consisting of 4,320 recall observations and 1,080 adversarial trap probes. SMART-SLOT achieved the highest recall accuracy, cross-session consistency, and demonstrated the greatest resistance to stale-fact prompts while with insignificant additional latency.
Erica Butts, Salam Daher· Proceedings of the 26th ACM...· 0 citations
The ability to handle long-term memory in LLMs is becoming increasingly critical, yet existing benchmarks remain English-centric and rely on aggregate retrieval metrics, failing to capture interactions between long-range context, temporal information, and reasoning. To address this, we introduce RUMBA (Russian User Memory BenchmArk) - a new benchmark for long-term conversational memory that provides a fine-grained taxonomy of memory-centric question types and a unified methodology accounting for semantic type, session scope, temporal reasoning, and the explicitness of temporal expressions. RUMBA consists of timestamped user-assistant dialogues with QA pairs requiring retrieval, combination, and reasoning across sessions. While designed for Russian, we also provide an aligned English subset under the same methodology. We evaluate contemporary memory systems and long-context models, and show how RUMBA serves as a diagnostic tool to analyze model behavior across benchmark slices and identify strengths and failure modes of different memory mechanisms.
E.D. Shevtsova, Inna Glebkina, Mark Baushenko et al.· arXiv.org· 0 citations
Memory systems for conversational LLMs are conventionally evaluated by direct, fact-seeking questions about prior dialogue (Direct QA): can the model recall fact X from a prior conversation? We tested whether higher Direct QA accuracy correlates with higher user satisfaction in a 4-month deployment (40 users, 1,872 sessions, 7 memory conditions). Existing-benchmark Direct QA varies from 19.7% to 70.1% across the 7 conditions, but satisfaction does not change. We hypothesize that existing benchmarks and user satisfaction are tracking different capabilities: benchmarks measure elicited retrieval (recall when asked), while conversation requires natural integration (detecting relevance and naturally weaving prior context into a response). To examine this, we introduce MemUse, a set of real user-cued memory moments drawn from the deployment, scored by an integration-aware judgment of the natural conversational response. Holding the model and context fixed, the same system that scores 78.8% on Direct QA references only 7.9% of those facts in conversation -- a 71-point gap. Within these moments, Natural Integration is associated with satisfaction, whereas Direct QA is not. We release the deployment corpus and MemUse together with all judgments and scoring prompts at https://github.com/ryuichi-sumida/memuse.
Ryuichi Sumida, K. Inoue, Tatsuya Kawahara· 0 citations
The results indicate that for precise, evidence-grounded questions over chat archives, much of the benefit credited to elaborate memory structures is recoverable by giving an agent controllable search over the unmodified record, with no LLM-based index construction at all.
Ruizhe Li, L. Zhang, Benfeng Xu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
