Finding Moments Worth Remembering: Multimodal Evidence and Multi-Stage Reasoning for Illustrating Actual-Play TTRPG Sessions
Abstract
Tabletop role-playing games present a challenging domain for artificial intelligence, particularly for reasoning about emotional shifts and narrative significance in long-form actual-play sessions. Players improvise shared stories over hours of dialogue, performance, rules negotiation, and social reaction, while the significance of memorable moments may depend on context beyond the local passage. We propose a hierarchical multimodal architecture for selecting emotionally and narratively significant moments as candidates for visual representation. The approach extracts speech, speaker, affective, conversational, and table-response evidence, contextualizes these signals through short-, medium-, and long-term narrative memory, and uses a multi-stage LLM-based judge to evaluate local, scene, and memory relevance before visualization. We evaluate moment selection using silver labels derived from viewer signals, and assess generated visual memories through human judgments of image--audio alignment. The system recovers referenced timestamps above a chance baseline at exact-window resolution, indicating mild alignment with viewer-perceived salience. Human evaluation indicates that the pipeline produces grounded images for well-chosen moments and that these images are preferred over those generated directly from transcripts. Together, these results represent a first step toward memory-supported visual externalization in long-form collaborative storytelling.