Subset Heterogeneity in Reward Model Benchmarks: A CONFIRM Validation of 174 RewardBench 2 Models
Abstract
Background. RewardBench 2 aggregates pairwise preference accuracy across heterogeneous task families (factuality, instruction following, safety, and others). A single leaderboard score can mask systematic subset specialization, yet standard benchmark reporting rarely tests whether accuracy is independent of task category. Methods. We applied CONFIRM, a chi-square test of independence with Cramér's V effect sizing and empirically anchored letter grades, to 174 publicly released reward models. For each model we constructed a 5 × 2 contingency table (subset × correct/incorrect) from published per-prompt scores (n = 1,763 prompts after excluding the non-binary Ties subset). The null hypothesis was that correct/incorrect outcomes are independent of subset category. Grades A–F reflect V magnitude (CONFIRM v2 thresholds); grade I denotes insufficient power. Non-significant results grade F only when power ≥ 0.80 to detect V = 0.10. Results. All 174 models rejected independence at α = 0.05 (all p < 0.02; median V = 0.272, range 0.082–0.467). No model received F or I. Grade distribution: A 36.8% (n = 64), B 50.6% (n = 88), C 11.5% (n = 20), D 1.1% (n = 2). Pooling across models, mean per-subset accuracy was lowest on Precise IF (36.3 per 100) and highest on Safety (75.7 per 100). In 93.1% of models the largest subset gap involved Precise IF as the weakest category; Safety was the strongest endpoint in 70.7% of cases. Conclusions. Published RewardBench 2 reward models exhibit statistically detectable subset heterogeneity at this sample size. Aggregate accuracy therefore understates structured performance imbalance, particularly weakness on precise instruction-following relative to safety-oriented subsets. CONFIRM provides a reproducible heterogeneity diagnostic to be read alongside ranking metrics. Plain-language summary. For all 174 reward models tested, the rate of correct judgments differed across the five task types by more than the prespecified statistical threshold. The direction of the difference was shared: 93.1% of models were weakest on precise instruction-following, and 70.7% were strongest on safety. Each model was tested on 1,763 prompts, enough sensitivity to detect even a small difference had one been present, so a model showing no difference would have been identifiable as such. None did. A single overall leaderboard score does not show this — two models with the same average can differ substantially underneath. This analysis measures whether a model's accuracy is uneven across task types, not which model is best overall, and it identifies a pattern without establishing its cause. Supplementary material. The deposited archive (rewardbench2_validation.zip) contains per-model results, subset breakdowns, contingency cell counts, validation flags, and step-by-step mathematical derivations for all 174 models. Competing interests. The author is affiliated with TraceSeis, Inc., which is developing CONFIRM as a commercial product. This constitutes a competing interest. All results are reproducible from the cited public data and the deposited analysis outputs. AI use disclosure. Generative AI tools were used during preparation of this work, in two distinct roles. For drafting and implementation: Anthropic Claude assisted with manuscript prose; the CONFIRM engine and analysis pipeline were implemented with AI coding tools (Cursor, Anthropic Claude) to the author's specification; and Google Gemini was consulted during writing and analysis runs. For review: Perplexity provided editorial review of a late draft, and xAI Grok was used as a general consistency check. This reflects the author's record of tool use and is not offered as an exhaustive log. Research design, statistical methodology, and interpretation are the author's. Because the analysis software was AI-implemented, every reported statistic was independently recomputed from observed cell counts and checked against pipeline output before reporting; per-model derivations are deposited as confirm_math.html and can be checked by hand. The author takes full responsibility for the contents of this record.
