A Five-Step Multi-Dimensional Learnability Validation Protocol for Meningitis Clinical Risk Assessment
Abstract
Clinical machine learning is routinely judged on discrimination alone. A model that separates classes well is reported as successful, even when its probabilities are miscalibrated, its explanations point at variables with no measured association with the outcome, and nobody has checked whether the target can be learned from the available features at all. This study proposes a five-step multidimensional learnability validation protocol that tests a prediction target on five axes: feature-target dependence before modelling, calibration through Brier score decomposition, clinical utility through decision curve analysis, interpretability through SHapley Additive exPlanations, and agreement between fitted coefficient signs and the directions reported in the clinical literature. The protocol is the contribution; no new learning algorithm is proposed. Five established classifiers, Logistic Regression, Random Forest, LightGBM, CatBoost, and XGBoost, are used as instruments on a public synthetic meningitis dataset of 1,200 records with 10 clinical predictors and three prediction targets. Under repeated stratified cross-validation with five folds and five repeats, the four tree-based models reached mean areas under the curve of 0.969 to 0.971 and mean Matthews correlation coefficients of 0.780 to 0.793 for aetiological diagnosis, and 0.969 to 0.972 and 0.791 to 0.802 for clinical risk level; both targets passed all five validation steps. The mortality target failed all five: the strongest feature-target correlation was 0.067 and the strongest mutual information 0.013, the Brier score of the Platt-scaled reference model (0.0949) did not fall below the uncertainty baseline (0.0944), no threshold between 0.05 and 0.50 gave net benefit above both default strategies, the leading SHAP attribution fell on a variable with no measured association, and only two of five literature-referenced coefficient signs agreed. Judged only by whether mean discrimination exceeded chance, mortality could have appeared marginally viable. The protocol is demonstrated, not externally validated, and a target that passes is eligible for further validation rather than ready for clinical use.