A comprehensive study on multi-dimensional predictive safety analysis of machine learning models in diabetes detection

Abstract

High discrimination performance is often interpreted as evidence of model reliability in clinical machine learning (ML). However, strong predictive accuracy does not guarantee safe decision behavior. In safety-critical domains such as healthcare, miscalibrated probabilities, ineffective uncertainty estimation, and overconfident errors can lead to harmful decisions despite high area-under-curve metrics. This study presents a comprehensive multi-dimensional predictive safety analysis demonstrating that models with comparable discrimination performance can exhibit substantially different reliability and risk characteristics. We conduct a rigorous benchmarking evaluation across two heterogeneous diabetes risk datasets, a large population-scale BRFSS cohort and a small-sample early-stage clinical dataset. A diverse set of classical, ensemble, and deep tabular models are evaluated under a strict leakage-free cross-validation protocol with dedicated calibration splits. Beyond conventional discrimination metrics (PR-AUC, ROC-AUC), we assess probabilistic calibration (ECE, Brier score, NLL), epistemic uncertainty using a unified bootstrap-based framework, uncertainty–error correlation, selective prediction safety, high-confidence error rates, subgroup fairness across age and sex, and conformal prediction with finite-sample marginal coverage guarantees under the standard exchangeability assumption. Our results reveal a consistent accuracy–safety gap that models achieving near-identical PR-AUC display markedly different calibration quality, uncertainty effectiveness, subgroup reliability, and overconfidence behavior. On the early-stage clinical dataset, near-ceiling discrimination performance is observed across models, but due to the limited sample size, these results should be interpreted cautiously, as variability across folds may affect the stability of comparative performance across models. The findings demonstrate that discrimination metrics alone are insufficient to characterize reliable clinical decision-making. A comprehensive evaluation incorporating calibration, uncertainty quality, subgroup fairness, and formal coverage guarantees is essential for the trustworthy pre-deployment assessment of machine learning in healthcare. Rather than introducing a new predictive algorithm, this work contributes a unified empirical audit of safety-relevant model behavior, addressing reliability dimensions that are commonly evaluated separately or omitted in conventional diabetes prediction benchmarks.