How Much of a Leaderboard Ranking Survives Its Own Sampling Error? A Paired-Resolution Audit of Five AI Benchmarks
A benchmark leaderboard is read as a stopwatch: the rank beside a model is treated as a fact. Yet every entrant is scored on the same items and publishes which it solved, so a leaderboard is a paired experiment, and its ordering can be tested rather than trusted. We test each adjacent-rank pair for statistical separabi...