Measurement reliability bounds functional benchmarks and relocates where variant effect prediction fails

Abstract

Background Variant effect predictors are increasingly benchmarked against multiplexed assays of variant effect (MAVEs) rather than clinical labels, which removes label circularity but introduces a new problem: a correlation against a measurement cannot exceed the measurement’s own reproducibility, and precision varies sharply across the territories compared. Results We scored nineteen predictors across sixteen strata of a frozen atlas of 64,178 saturation genome editing variants in seven cancer-susceptibility genes. From published replicate scores and standard errors we estimated each territory’s reliability ceiling, and showed by simulation that the correction reduces error above a ceiling of about 0.45 and amplifies it below. Ceilings vary more across territory than predictors do, and correcting for them redraws the map at the splice extremes. The collapse at canonical splice sites is largely a property of the assay: the median shortfall relative to coding narrows from 1.7-to 1.4-fold; this convergence survives dropping BARD1 or PALB2 but inverts when BRCA1 is dropped, so we report all three leave-one-gene-out folds rather than claim gene independence, and the frontier parity rests on one deposit. Genuine failure lies 11–50 bp into the intron, which the uncorrected map presents as modest. Across MaveDB, 2,452 of 2,803 score sets carry, at the upper bound, what a reliability estimate needs, though a conventional column-name search finds only a tenth; among 674 human deposits with a computable ceiling, 29.9–51.8% fall below 0.90. Scored as classification against the assays’ own functional calls in three genes, the same predictors separate damaging from tolerated better than their correlations suggest, though none reaches the strongest evidence band at the 95%-specificity operating point. Conclusions Territory-resolved benchmarks should report a per-stratum reliability estimate, or state that the assay permits none. It asks nothing of depositors and applies today, at the upper bound, to most (87%) of MaveDB.