Comparing human narrative engagement with model attention mechanisms suggests explanations for degraded narrative comprehension and targets for future development.
Abstract
Although LLM context lengths have grown, there is evidence that their ability to integrate information across long-form texts has not kept pace. We evaluate one such understanding task: generating summaries of novels. When human authors of summaries compress a story, they reveal what they consider narratively important. Therefore, by comparing human and LLM-authored summaries, we can assess whether models mirror human patterns of conceptual engagement with texts. To measure conceptual engagement, we align sentences from 150 human-written novel summaries with the specific chapters they reference. We demonstrate the difficulty of this alignment task, which indicates the complexity of summarization as a task. We then generate and align additional summaries by nine state-of-the-art LLMs for each of the 150 reference texts. Comparing the human and model-authored summaries, we find both stylistic differences between the texts and differences in how humans and LLMs distribute their focus throughout a narrative, with models emphasizing the ends of texts. Comparing human narrative engagement with model attention mechanisms suggests explanations for degraded narrative comprehension and targets for future development. We release our dataset to support future research.
Long-form video understanding encompasses tasks that go beyond retrieving isolated events, including tracking an evolving narrative and interpreting social meaning that may remain implicit. However, existing benchmarks rarely evaluate these capabilities jointly, particularly in high-context, non-English media. To address this gap, we introduce NARU, a benchmark designed to evaluate Narrative evolution and Reasoning on cultural Understanding in Japanese long-form video. NARU consists of 1,481 questions grounded in 155 videos totaling 146.8 hours, spanning four narrative and five cultural dimensions. To construct the benchmark at this scale, we propose a hierarchical memory-based annotation pipeline that transforms raw video into structured event, narrative, and cultural annotations, then generates questions via task-oriented synthesis and iterative shortcut removal. The construction process includes two native-speaker verification stages involving 68 annotators. Evaluations across eight model configurations reveal substantial limitations in both long-range narrative integration and culturally grounded reasoning. By exposing these persistent gaps, NARU offers a systematic testing ground for developing MLLMs capable of reliably interpreting long-form, high-context video.
Yuheng Huang, Jianlang Chen, Jiayang Song et al.· 0 citations
Large language models have advanced long-form story generation, yet the resulting narratives often fail to preserve facts, states, and relationships established earlier in the same text. This survey examines the problem as narrative consistency, defined as the task-conditioned preservation of binding propositions in the operative narrative state. Using a targeted evidence-mapping strategy, we analyze 90 papers, with a literature search cutoff of April 23, 2026, covering long-form story generation, narrative evaluation, consistency detection, and mitigation strategies. We distinguish narrative consistency from factuality, faithfulness, surface coherence, and hallucination by centering the dynamically accumulated evidence of the story itself. The survey organizes consistency judgments around three evidence sources, namely story-internal propositions, source-or-canon evidence, and external-world knowledge, as determined by task conditions that specify which source is binding. We introduce a four-category, fourteen-subtype taxonomy comprising World and Setting, Character-Agentive, Event-Structural, and Narration and Discourse categories. We use the taxonomy as a common reference frame for analyzing benchmark coverage, detection methods, mitigation strategies, and open challenges, highlighting where current work concentrates and where coverage remains thin. We also separate legitimate creative extension from task-inconsistent additions that contradict, revise, or exceed the permitted generation setting. The survey closes by identifying needs for evidence-grounded oracles, subtype-aware calibration, omission and discourse-level failure evaluation, and task-conditioned verification.
Keunhyeung Park, Seunguk Yu, Jinhee Jang et al.· IEEE Access· 0 citations
This study examines how different prompt anchoring strategies influence the conceptual representation of LLM-generated keywords and compares those effects with the effects of model selection. A controlled exploratory experiment evaluated four prompt conditions—No Examples, Brief Keywords, Detailed Explanations, and Author-Based Examples—across T. D. Wilson’s four information behavior dimensions using 1068 abstracts. Four LLMs (GPT-4o-mini, Claude-3-haiku, Gemini-2.0-flash-lite, and DeepSeek V3) were evaluated under all prompt conditions, yielding 17,036 valid observations. Results indicate that model identity accounts for substantially more variance in keyword generation (η2 = 0.309) than prompt condition (η2 = 0.069), although these estimates should be interpreted with caution given the repeated-measures design and assumption violations. Prompt anchoring, however, consistently reconfigured the conceptual distribution of outputs across all models, indicating that it influences conceptual representation even when model effects are larger. Author-Based Examples substantially increased representation of the typically underrepresented Information Sharing dimension, whereas Detailed Explanations produced the highest overall generation rates and the broadest dimensional coverage. These findings further indicate that different anchoring strategies involve consistent trade-offs in dimensional coverage. The study thereby identifies prompt anchoring as a source of methodological variation in LLM-assisted content analysis, indicating that anchoring strategies should be explicitly specified, justified, and reported as part of the study methodology.
Although large language models (LLMs) have demonstrated impressive creative fiction generation, they struggle to maintain narrative consistency and coherent plot lines in long-form stories. In this work, we introduce a unified framework for long-form narrative generation and verification. MAGNET, a multi-agent goal-driven narrative engine for storytelling, generates stories with persona-grounded character agents that propose actions based on a shared world state and evolving story goals, while ATLAS is a graph-based pipeline that compares scene-level world representations across a generated story to detect hallucinations. By evaluating MAGNET using an LLM editor, pairwise rubric scoring, and ATLAS, we show that our framework produces coherent narratives compared to single-model prompting and IBSEN. At 100 pages, MAGNET reduced annotations and hallucinations by 41 and 50%, respectively, compared to the single model baseline and by 34 and 45%, respectively, compared to IBSEN, with pairwise rubric evaluation showing similar results. These results suggest that long-form narratives can emerge from explicit world-state tracking and goal-driven multi-agent generation, providing a foundation for controllable and structurally coherent long-form narrative generation.
Aayush Aluru, C. Ho, Muhammad Hammouri et al.· 0 citations
We describe our system for the Narrative Similarity task at SemEval-2026 (Task 4), where the goal is to determine which of two candidate stories is more similar to an anchor story directly (Track A) or via vector representations (Track B). For Track A, our strategy leverages commercial, closed-source Large Language Models (LLMs) to generate multiple independent judgments per story triple. Simple majority voting provides strong performance in high-agreement cases, but it is unreliable when the judgments exhibit weak agreement. For difficult cases, we compare the stories along three narrative dimensions—theme, course of action, and outcome—prompting the LLMs to score similarity for each component on a scale of 1–4 and learning optimal combination weights on development data. We further find that chain-of-thought–style prompting with detailed reasoning outputs achieves comparable results to the scoring approach on difficult examples. We also conduct a data analysis revealing substantial annotation variability, which helps understand the difficulty of the task. Our system ranked 1 st in both tracks, achieving 0.78 test accuracy in Track A and 0.72 in Track B, where embedding only the course-of-action component yielded the best result.
Tisa Islam Erana, Azwad Anjum Islam, Anshu Kiran Sharma et al.· SemEval@ACL· 1 citation· ⚡1
