TruthLens: A Verification-Gated Pipeline for Evidence-Grounded Fact-Checking of Short-Form Political Video

Abstract

TruthLens is an evidence-first pipeline for fact-checking short-form political video and image posts. It decomposes a post into atomic claims, researches each against the open web, and subjects every generated verdict to a deterministic, non-LLM validator — checking citation existence, source-fetch success, numeric grounding, and label/reasoning consistency — before anything is published. This record contains the preprint, the TruthLens-202 evaluation corpus, and the supplementary results and protocols required to reproduce the reported numbers. Contributions A methodological correction, reported with the same prominence as any positive result. A post-hoc audit found that, across the entire program, the baselines had been given a clean human-written claim summary rather than the claim TruthLens's own extraction stage produced — inflating baseline performance. Corrected, on the frozen six-item paired comparison, full TruthLens reaches 33.3% (2/6) against 50.0% (3/6) for the strongest baseline and 0/6 for the other two. At this sample size the Wilson intervals overlap almost entirely (exact McNemar p = 1.0); the comparison is retained for the methodology lesson, not as an architecture ranking. A formally defined, deterministically checkable “support validity” construct — whether a verdict's citations, sources, numbers and label/reasoning consistency hold up against its own evidence matrix — with a single-sample observation (n = 9) that support validity and label accuracy move independently: two ground-truth-independent validator fixes moved validator recall from 1/6 to 2/5 while reel-level accuracy stayed unchanged. TruthLens-202, a 202-item evaluation corpus (9 dev + 193 validation) of real, publicly posted short-form political content across X/Twitter, Instagram and Facebook, distilled from 11,544 screened candidates across seven fact-checker archives. Every verdict is anchored to an independent professional fact-check by a short verbatim quotation. Released as URLs, labels and annotations only — no media, captions, transcripts or article bodies are redistributed. The first end-to-end evaluation on that corpus, in a zero-cost local-only configuration (every stage on a local 3B model, keyless search, no cloud escalation), reported explicitly as a floor: 34.7% of posts (67/193) produce a verdict at all; bucketed accuracy on those is 29.9% (20/67, Wilson 95% CI [20.2, 41.7]); balanced accuracy is 22.2%, below an always-FALSE predictor's 33.3%; macro-F1 is 0.194; the MISLEADING class is never recovered (0/7). Infrastructure failure was 0%. The transferable finding is diagnostic rather than performative: claim extraction — not retrieval — determines whether a post is checked at all, with verdict reasoning a second, smaller failure surface on the posts that do resolve. A 24-entry taxonomy of failure modes surfaced only by running the complete system against real content, all but three paired with a deterministic check and a regression test built from the case that surfaced it. Ground-truth provenance, stated on two axes Verdict provenance is uniform: all 202 verdicts are independent professional fact-checking organisations' published findings, anchored to a verbatim quotation. Annotation provenance is not: the 9 dev items were labelled directly from the fact-checker's own ClaimReview, whereas for the 193 validation items the claim phrasing, taxonomy mapping and promote/defer/reject decision were performed by a large language model under the author's direction and applied in batch, not drafted by a model and then adjudicated item by item. A machine-readable annotation_provenance field records this per item. Relatedly, the model that screened which of the 11,544 candidates entered the corpus was the same model later evaluated on it. Because that screen favours posts the model can parse, the selection bias runs toward easier items — so the reported accuracy is best read as a favourably-selected floor, not a pessimistic outlier. Limitations No inter-annotator agreement statistic exists and none is fabricated. 88% of the validation ground truth comes from one fact-checking organisation. The corpus carries three strong composition skews (83% FALSE, 83% X-sourced, 81% provenance-type) requiring balanced accuracy and per-class F1 rather than raw accuracy. 28 of 193 items sit outside the review protocol. Only one system configuration has been evaluated. Status Preprint. Not peer reviewed, not published in any journal or conference proceedings, and no indexing is claimed.