variant-fm-benchmark: ranking, calibration and evidence strength for splice-region variant effect predictors against a functional standard

Abstract

Ranking is saturated, calibration is not, and neither changes the evidence strength — code and frozen analysis set Software and frozen data underlying the manuscript "Ranking is saturated, calibration is not, and neither changes the evidence strength: an independent benchmark of splice-region variant effect predictors against a functional standard" (N. Zhang). WHAT CHANGED IN THIS VERSIONThis release adopts frozen-matrix-v2 as the analysis set. The SpliceAI, Pangolin and Nucleotide Transformer columns of the published frozen-matrix-v1 are replaced by the output of pinned scorers — Pangolin at git commit 5cf94b8 and SpliceAI at full floating-point precision, both reproducing the v1 columns exactly when re-rounded to the printed precision; Nucleotide Transformer re-scored under a pinned environment (rank correlation with the original column ρ = 0.9997) — and nothing else in the matrix is changed. The external TP53 validation now uses all ten predictors. Positive likelihood ratios are reported on two bases: at the scanned observed threshold and interpolated to exactly 95% specificity. The v1 result tables are retained beside the v2 tables, with a listing of every quantity that moved between them. CONTENTS- The analysis pipeline and the automated test suite.- The content-hash-pinned frozen matrix (frozen-matrix-v2, built from the published frozen-matrix-v1).- The per-gene official-label mapping.- A two-command rebuild (scripts/reproduce_calibration.py for the main analyses, scripts/reproduce_robustness.py for the robustness analyses).A clean-clone re-run reproduces every tracked result table byte for byte; the reproduction commands verify both content hashes before any analysis runs and stop if either does not match. All seeds are fixed. DATA PROVENANCEThe functional-standard scores are third-party public deposits, cited by accession and not redistributed here: MaveDB urn:mavedb:00000097-0-2 (BRCA1), 00001225-a-1 (BRCA2), 00001250-a-2 (BARD1), 00001259-a-2 (PALB2), 00000673-0-1 (RAD51C), 00000675-a-1 (VHL) and 00000662-0-1 (BAP1). Every predictor score was produced by the cited tool under a pinned environment. RELATEDCompanion deposit (measurement-reliability atlas), all versions: https://doi.org/10.5281/zenodo.21828448Manuscript preprint: [bioRxiv DOI — add after posting] Use of large language models: large language model assistance was used to write analysis code and in preparing the manuscript. No data were generated by the model; every functional score is a third-party deposit and every predictor score was produced by the cited tool under a pinned environment. LICENSING — PLEASE READ BEFORE REUSEThe licence field of this record (MIT) covers the code only. The functional measurements are third-party MaveDB deposits: BRCA1, BRCA2, BARD1 and PALB2 under CC0 1.0; RAD51C, VHL and BAP1 under CC BY 4.0. The ten predictor score columns carry the terms of the tools that produced them, and five of the ten are restricted to non-commercial use: AlphaGenome (non-commercial, API free tier), SpliceAI (CC BY-NC 4.0 for the trained models), CADD (non-commercial), AlphaMissense (CC BY-NC-SA 4.0, non-commercial and ShareAlike) and Nucleotide Transformer (CC BY-NC-SA 4.0, non-commercial and ShareAlike). GPN-MSA (MIT), phyloP-100way and phastCons-100way (UCSC track data) and gnomAD allele frequencies (CC0 1.0) require attribution at most; Pangolin is GPL-3.0 software, and running it does not place its output under the GPL. Each restriction travels with its column and with any figure or table derived from it. The per-column determinations are in this archive's LICENSE.