PREreview of "What AI Benchmarks Actually Measure: Adapting Convergent and Discriminant Validity to Interrogate Fifty-Six AI Benchmarks"

Abstract

This Zenodo record is a permanently preserved version of a PREreview. You can view the complete PREreview at https://prereview.org/reviews/23180288. ## Summary Benchmark scores shape which models get funded, bought, and regulated — yet whether benchmarks measure what they claim to measure is rarely tested. This paper imports convergent and discriminant validity from psychometrics and applies them at unprecedented scale: 56 capability and safety benchmarks, 53 models from 31 families, with model outputs and scores released at item and benchmark level. Four findings emerge. First, safety benchmarks with the same assigned concept often correlate weakly with each other. Second, capability benchmarks correlate about as strongly across concepts as within them. Third, benchmarks sharing design elements (score format, task structure) sometimes correlate more strongly than benchmarks sharing an assigned concept — what a benchmark measures may be driven by how it's built, not what it claims. Fourth, individual benchmarks can measure something other than their purported concept: BBQ-accuracy, which claims to measure bias, correlates more strongly with reasoning benchmarks than with other bias benchmarks — and since BBQ is often the only bias benchmark reported in commercial model releases, that leaves a significant evaluation gap. ## Strengths This is the paper the benchmark-reform conversation has needed: one of the first attempts to measure construct validity at scale, with a real dataset and a principled framework rather than case studies. The design-elements finding is the most consequential — if score format predicts model rankings better than the purported capability does, then a large fraction of "capability" benchmarking is measuring test-taking format. The BBQ result is a genuinely important public finding, stated plainly without sensationalizing. The methodological honesty about the framework's own limits increases rather than decreases trust in the findings. Releasing the full item-level dataset turns the paper from an argument into infrastructure. ## Major concerns and questions 1. Everything rests on the assigned concepts — the authors' own labeling of benchmarks into shared concepts. Noisy labels cut both ways: weak convergence might mean the benchmarks are inconsistent, or the assignment lumped together things that shouldn't be lumped. The paper needs validation of the labeling step itself: inter-annotator agreement, sensitivity analysis, or a published annotation guide. 2. The general-capability confound: some models are just better at everything, and the framework can mistake a general capability factor for either convergence or its absence. A factor-analytic control — or how much variance the first principal component explains — would help readers judge whether they're looking at validity structure or just model quality. 3. Scope exclusions (non-English, human-evaluated, non-public, multi-turn) remove arguably the slice of the ecosystem where validity concerns are worst. The title's claim should be scoped explicitly. 4. BBQ: an alternative reading is that bias tasks require reasoning, so capability dominates the ranking even if the benchmark genuinely probes bias. The practical implication is the same (don't rely on BBQ alone), but the conceptual distinction matters for how benchmark builders respond. 5. All outputs were collected zero-shot at temperature 1. The headline findings should come with a stated robustness check: do the convergence/discrimination patterns survive greedy decoding? ## Overall assessment An important, field-level paper that moves benchmark criticism from complaint to measurement. The dataset alone is a major contribution. The analysis would be stronger with validation of the concept-labeling step and a direct confrontation of the general-capability confound — both addressable in revision. **Verdict: Recommend with revisions.** Competing interests The author declares that they have no competing interests. Use of Artificial Intelligence (AI) The author declares that they used generative AI to come up with new ideas for their review.