CheckThat! 2026 Task 3 asks systems to generate fact-checking articles, graded by an unweighted mean of four sub-metrics (M4). Our UTS submission placed 2nd of 11 teams (M4 = 0.484). The shipped system is a deterministic stub drafter wrapped by two single-lever interventions: a domain-attribution cite frame (HostCite) and a shadow-validated anchor picker (ShadowVal) that use Llama-3.2:1B only as a per-cite validator, never as a body-prose generator. The stack lifts M4 by +0.027 over the stub on the WatClaimCheck validation split, beats the field on entailment and coverage, and follows two design rules our ablation matrix made unambiguous. Scorer conservatism: credit only tokens the references entail - templates pay; LLM prose, reviewer names, and raw evidence all fail. Auxiliary anchor signals are miscalibrated against the Llama judge: every anchor proxy we tried (cross-encoder, length, lead position) picks anchors the judge rejects - gate on the judge itself. The remaining +0.062 gap to the winner sits on citation precision/recall (0.299 vs 0.671), consistent with a selective-emission policy that drops low-confidence cites.
Grievance is one of the warning signs analysts look for when assessing threats of violence. It is increasingly measured at scale from online text, most often with word-level lexicons like the Grievance Dictionary that score by matching weighted terms. Such matching is a fast and transparent proxy, but it cannot resolve whether a term is asserted, quoted, negated, or condemned. These lexicons are also often evaluated on pools enriched with the very examples they retrieve, so a high score partly reflects agreement with the lexicon's own selection rule. Examining a five-language, 2{,}000-item evaluation pool, we find its halves separated almost perfectly by the lexicon itself: every item labeled ``random''is in fact lexicon-negative, so the lexicon's apparent macro-AUROC of 0.686 collapses to a 0.500 floor fixed by construction. We keep the dictionary's 22-construct ontology but replace term matching with context-reading models, evaluated on a non-circular benchmark that separates unconditional-random, lexicon-positive, and lexicon-negative strata across five languages. Reading the full post rather than the target sentence alone helps most where the lexicon is silent, raising average precision on lexicon-negative text from 0.14 to 0.20, with the largest gains on quoted, implicit, and cross-sentence grievance. Together, these results show that grievance is measured more faithfully by reading the surrounding context, and more honestly when tested on text the lexicon did not select. We release our code and benchmark at https://github.com/behavioral-ds/multilingual_grievance.
Lin Tian, M. Rizoiu· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
