ARIC: A Cognitive Framework for Explanatory Narrative Evaluation in Conversational Information Seeking Systems
Conversational information seeking (CIS) systems now generate explanations, but we still evaluate them using retrieval-focused metrics such as faithfulness, completeness, and source attribution. These checks are necessary, but they do not tell us whether a response helps users form a coherent mental model. Cognitive science treats understanding as an active construction guided by causal structure, coherence, and the organization of information. Evaluation should therefore test whether explanations support integration, inference, and retention. This paper aims to define the target form of communication and a way to assess it. We introduce Explanatory Narratives for CIS, which combine the organizing benefits of storytelling to enable explanation. We then propose ARIC, a cognitively grounded framework for evaluating explanatory narratives across four comprehension stages: Attention, Representation, Integration, and Consolidation. To demonstrate its value, we apply ARIC to human-authored explanatory narratives and show how stage-based analysis yields actionable diagnostic insights. This shifts CIS evaluation toward the question the IR community increasingly faces: whether system-generated explanations actually help users understand.