Beyond Marginal Coverage: Class-Conditional Conformal Prediction Reveals Diagnosis-Specific Uncertainty Structure in Psychiatric Neuroimaging

Abstract

Conformal prediction is increasingly proposed for clinical decision support because it provides distribution-free coverage guarantees that hold for any model and any data distribution. The guarantee routinely reported, however, is marginal, and we show that in multi-class diagnostic classification it can be satisfied exactly while individual diagnoses are covered very unevenly. On a four-way mood and psychosis classification task over 1,520 subjects from three studies and 14 acquisition sites, marginal split-conformal calibration attained 0.9000 empirical coverage against a nominal 0.90, while healthy controls were covered at 0.941 and schizoaffective disorder at 0.819. The surplus and the deficit cancel, so the aggregate figure is uninformative about either. Class-conditional (Mondrian) calibration reduced this 12.2-point disparity to 0.4 points at a cost of 0.07 labels in mean set size, under 3%. Holding coverage fixed across diagnoses has a further consequence: the residual variation in set size can no longer be attributed to class prior or per-class accuracy, and becomes interpretable as a property of the subject. We show that the resulting prediction sets separate subjects into confident, boundary, ambiguous and unresolved strata, and that the proportion of subjects independently flagged as label-ambiguous by a structural-MRI model trained separately on the same cohort rises monotonically across these strata (34.7%, 57.0%, 68.0%, 82.1%; p = 2.1 × 10−17). No schizoaffective subject and 0.9% of bipolar subjects reach the confident stratum, against 18.4% of controls and 13.0% of schizophrenia subjects. Set size at matched coverage also provides a comparison between representations that accuracy cannot. Applied to structural MRI, functional MRI and their fusion, it reveals a sign reversal that accuracy conceals: fusion reduces set size for bipolar (−0.25) and schizoaffective (−0.23) subjects and increases it for controls (+0.16), and the aggregate fusion benefit itself changes sign below α = 0.10, while top-1 accuracy rises monotonically from 0.495 to 0.578 to 0.611 across the three models. Reporting marginal coverage alone is insufficient whenever diagnostic classes are unbalanced.