How Much of a Leaderboard Ranking Survives Its Own Sampling Error? A Paired-Resolution Audit of Five AI Benchmarks

Abstract

A benchmark leaderboard is read as a stopwatch: the rank beside a model is treated as a fact. Yet every entrant is scored on the same items and publishes which it solved, so a leaderboard is a paired experiment, and its ordering can be tested rather than trusted. We test each adjacent-rank pair for statistical separability - exact McNemar's test on discordant items for pass/fail benchmarks, a paired item-bootstrap for scored ones - across five public boards. On SWE-bench Verified, 129 of 133 adjacent ranks are not separable at the 5% level; on MTEB, 176 of 180; on HELM Lite, all 89 of 89. The benchmarks are not broken: 88% of all pairs separate cleanly. It is the neighbours they cannot order, and neighbours are what the top of a table is made of. We derive a resolution bound, delta >= 2.80*sqrt(d/n) in the discordance rate d, that predicts which pairs are decidable from a benchmark's size, validate it on 12,739 pairs with zero false positives, and use LMArena - which already ships confidence intervals and ties - as a positive control for what correct reporting looks like. Every figure is recomputed from public data and passes a self-test suite; the recomputed rates match each official board to its published precision, with one mismatch on the Test split that reproduces a known duplicate-id defect in a submission file. We argue that a board should report the resolution of its ranking alongside the ranking, at no additional data cost. Version 2 (18 September 2026). No measurement changed: every quantitative result was rerun from the deposited data and reproduced, including the three parity gates, the preregistration scoring and all four figure coordinates. What changed are sentences about other people's work. The result that dropping a small share of preference data can change Chatbot Arena's top model belongs to Huang, Shen, Wei and Broderick, not to Singh et al., whose paper documents private testing and unequal data access instead; Zhang and Hardt measure a diversity-stability trade-off rather than proving an impossibility; Card et al. (2020) and Mogstad et al. are now cited as the prior art for the power requirement and for simultaneous rank confidence sets; the citation to Lunardi, Della Mea, Mizzaro and Roitero named only its last two authors and had strengthened their hedges; LMArena's published ranking rule is one plus the number of models significantly above, which reproduces its ranking for all 242 models, and its ten rank-2 models sit at rating positions 2 to 9, 11 and 20. Four checkable claims about the author's own artefacts are corrected in the same pass. All of it is listed in a new section, "Corrections in this version". Version 3 (18 September 2026). Version 2 corrected twelve statements about other people's work; a re-audit of it found that the correction had been incomplete in one direction and wrong in another. Three items reached version 2's deposit description but never the paper itself: the abstract still claimed the recomputed rates match each official board exactly, where the results file records one mismatch of 1.22 pp on the Test split, traceable to a submission file listing 241 results over 213 distinct instance ids; the per-split agreement figures were listed in a different order from the table above them; and the abstract still derived the bound from benchmark size alone rather than from size and near-pair discordance. The Mogstad et al. entry added in version 2 carried the wrong year, a truncated title and no venue, which is the defect the correction existed to remove. And Table 2 carried a column headed "#1 survives reshuffle" whose values, "every time" and "never", had no computation anywhere in the repository. That check is now defined and run (top1_stability.py, deposited here): over 2,000 item bootstraps the leader holds on Test 100% of the time, on Verified 38.3% and on Lite 53.6%, so "never" was wrong as well as unevidenced, and the Multimodal split, which has no per-instance matrix, now reads "not computed". Version 4 (19 September 2026). One reference. The Huang et al. entry gave the title as "Dropping Just a Handful of Preferences Can Change Top LLM Rankings"; it is "Top Large Language Model Rankings", and the entry now names all four authors. The same truncation was corrected in a companion paper on 18 September, and the letter sent to the first author that day said it was fixed, which was true of one deposit and not of this one. Nothing else changed.