Skip to content

SpeakerMem-R1: Speaker-Centered Dual-Track Memory for Multi-Party Dialogue

Sep 2026 · arXiv.org · Vol abs/2609.26780 · 0 citations · 88 references
Computer Science

TL;DR

SpeakerMem-R1 is proposed: its dual-track memory stores speaker-labeled verbatim messages and derived states organized into person-level and group-level views, then combines evidence from both tracks by entity, event, and time at query time, and ablations show that the verbatim and structured tracks, as well as person-level and group-level views, are complementary under the standardized evaluation interface.

Abstract

Long-term conversational memory in multi-party settings requires more than retrieving relevant content from long-term conversations: it must distinguish who said what, whom each statement concerns, how individuals perceive one another, what information is shared by the group, and how states change over time. Recent studies on multi-party dialogue benchmarks show that existing general-purpose LLM memory systems tend to lose person and group relations or struggle to integrate clues distributed across members, groups, and time. Together, these issues reveal two core bottlenecks: message attribution and relational understanding in multi-party dialogue, and state reconstruction from interleaved histories. To address both, we propose $\textbf{SpeakerMem-R1}$: its dual-track memory stores speaker-labeled verbatim messages and derived states organized into person-level and group-level views, then combines evidence from both tracks by entity, event, and time at query time. To reduce attribution and update errors during structured memory construction while enabling local deployment, we train Writer-R1 with SpeakerLevenshtein and speaker-conditioned GRPO. On GroupMemBench, SocialMemBench, and EverMemBench, SpeakerMem-R1 achieves binary accuracies of 47.9%, 69.2%, and 61.9%, respectively. On the publicly reported EverMemBench leaderboard from EverMind-AI, we achieves 62.33%, the best reported result among the latest state-of-the-art frameworks. It also achieves 70.85% on all 1,986 LoCoMo questions, which we use as a two-person long-term conversation boundary test. In a controlled evaluation of 305 questions, RL raises the SFT Writer's mean accuracy from 57.38% to 68.20%. We report both binary accuracy and token-F1, and ablations show that the verbatim and structured tracks, as well as person-level and group-level views, are complementary under the standardized evaluation interface.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

Beyond Dyadic Memory: Interaction-Aware Multimodal Memory with Adaptive Agentic Retrieval for Multi-Party Spoken Conversations

This work proposes VoxPolyMem, an interaction-aware multimodal memory framework combining incremental speaker identification with a memory hierarchy comprising interaction memory, fact memory, and participant profiles, and introduces Evidence-Gain GRPO (EG-GRPO), which uses round-wise credit assignment to encourage com...

Wen-Xu Jia, Xi-Ze Cheng, Zi-Han Zhang et al. · 0 citations
#natural language process... Book Open access Sep 2026

TEIDAN: A Multilingual Multiparty Dialogue Corpus

This paper describes the collection design, participants, recording setup, transcription format, and corpus statistics, and provides preliminary analyses to illustrate how TEIDAN can support research on turn-taking, addressee recognition, and multimodal grounding in human-human and human-agent interaction.

Taiga Mori, K. Inoue, Mikey Elmers et al. · 2 citations

Who Said What, and Will It Be Remembered? Evaluating Persistent Speaker Attribution Across Meetings

Speech transcripts used as long-term memory must preserve both words and stable speaker identities. Existing meeting-transcription metrics either ignore speakers or remap anonymous speakers independently in each recording, so they cannot measure whether the same person retains one identity across meetings. We evaluate...

S. Vispute, A. M. Mishra, Siddhartha Saxena · 0 citations
Sep 2026

Voices as Handles: Reasoning about Speaker Identity with Frozen Text LLMs

The proposed Speaker Handles provide an efficient way to integrate acoustic speaker identity into frozen text LLMs for speaker-content reasoning, and are presented as a controlled shared-agent benchmark in which overlapping facts across users require correct cross-session speaker attribution.

Run-Qiu Xu, Zhi-Sheng Zheng, David Harwath · 0 citations
Preprint Aug 2026

FTA-Mem: Fact-Time-Affect Anchored Memory for Low-Density Long-Term Dialogue

FTA-Mem is proposed, a structured memory framework for low-density long-term dialogue that uses Boundary-preserving Window Segmentation to form coherent situation fragments, and constructs Fact-Time-Affect Memory Units that jointly encode factual content, temporal grounding, and affective context.

Chang Liu, Jun Zhao, Shu-Yi Zhang et al. · 0 citations
Preprint Aug 2026

MemUse: Moving Memory Evaluation from Direct QA to Natural Integration in Long-Term Human-AI Conversation

It is hypothesize that existing benchmarks and user satisfaction are tracking different capabilities: benchmarks measure elicited retrieval (recall when asked), while conversation requires natural integration (detecting relevance and naturally weaving prior context into a response).

Ryuichi Sumida, K. Inoue, Tatsuya Kawahara · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.