A methodological framework for real-world performance studies of clinical variant classification platforms at early organizational stages

Abstract

Automated platforms that classify sequence variants under the American College of Medical Genetics and Genomics/Association for Molecular Pathology (ACMG/AMP) framework are increasingly used in rare-disease genomics, yet methodology for characterizing their performance at an early organizational stage is underdeveloped. The two framings most common in the literature are poorly suited to this stage: single-source comparison treats one peer laboratory or curated set as ground truth, which is difficult to defend at evidence depths where qualified laboratories disagree 25%–35% of the time; and formal regulatory adjudication requires multi-adjudicator panels and quality-management infrastructure that small or single-founder developers do not yet have. This paper proposes a methodological framework for Real-World Performance Studies that occupies the space between informal in-house testing and formal regulatory adjudication. The framework has four components: a three-layer performance model that separates analytical, classification, and clinical performance and forces every observation to a locus of attribution; a multi-source ground-truth construction with an explicit, evidence-strength-ordered weighting hierarchy; a six-category methodological-disposition taxonomy that resolves platform-versus-comparator disagreement into characterized categories with distinct action implications, only one of which denotes a classifier defect; and a phased pathway that carries early-stage evidence forward toward eventual regulatory submission. The framework is demonstrated, not validated, on three real-world cohorts (97 scored cases across three classifier versions) of a variant classification platform developed by Helena Bioinformatics. The demonstration illustrates how the method is operationalized. It does not test, validate, or claim superiority for any system. The framework is non-proprietary and is offered for independent adoption and adaptation. Its principal limitations - single-platform demonstration, single peer comparator, a taxonomy developed on the same cohorts that illustrate it, and structurally challenging conflicts of interest - are stated explicitly and are the reason the paper claims a method, not a result. The proposed framework is a conceptual methodology operationalized as a structured human-in-the-loop protocol; it is not a turnkey software package, and empirical use requires version-locked automated outputs plus documented expert review.