COGNAC at SemEval-2026 Task 4: Evaluating Narrative Components with LLMs for Hard Story Similarity Cases
TL;DR
This system for the Narrative Similarity task at SemEval-2026 (Task 4), where the goal is to determine which of two candidate stories is more similar to an anchor story directly or via vector representations, finds that chain-of-thought–style prompting with detailed reasoning outputs achieves comparable results to the scoring approach on difficult examples.
Abstract
We describe our system for the Narrative Similarity task at SemEval-2026 (Task 4), where the goal is to determine which of two candidate stories is more similar to an anchor story directly (Track A) or via vector representations (Track B). For Track A, our strategy leverages commercial, closed-source Large Language Models (LLMs) to generate multiple independent judgments per story triple. Simple majority voting provides strong performance in high-agreement cases, but it is unreliable when the judgments exhibit weak agreement. For difficult cases, we compare the stories along three narrative dimensions—theme, course of action, and outcome—prompting the LLMs to score similarity for each component on a scale of 1–4 and learning optimal combination weights on development data. We further find that chain-of-thought–style prompting with detailed reasoning outputs achieves comparable results to the scoring approach on difficult examples. We also conduct a data analysis revealing substantial annotation variability, which helps understand the difficulty of the task. Our system ranked 1 st in both tracks, achieving 0.78 test accuracy in Track A and 0.72 in Track B, where embedding only the course-of-action component yielded the best result.