Explainable Semi-Supervised Learning Framework for Alzheimer’s Disease Prediction Using SHAP-Based Feature Selection and Cost-Sensitive CatBoost

Abstract

Alzheimer’s disease (AD) remains a critical global health challenge for which early diagnosis is essential for effective intervention. However, AD detection is a challenging and complex task due to the scarcity of labeled clinical data, class imbalance, and lack of prediction interpretability in healthcare systems. Although current machine learning frameworks are promising, they are frequently based on fully supervised environments and lack transparency, which limits their ability to be adopted in real-world clinical scenarios where labeled data are often scarce. This study proposes a new lightweight, semi-supervised, SHAP-enhanced CatBoost (LSSSC) framework that covers the challenges related to insufficient label data, interpretability of prediction models, and performance in a single learning paradigm. The proposed solution provides two major innovations: (i) a hybrid semi-supervised learning algorithm that incorporates the use of confidence-aware pseudo-labeling with cost-sensitive learning, allowing the effective use of the unlabeled data and alleviating the problem of class imbalance, and (ii) an intrinsic SHAP-based feature selection mechanism that assists not only in increasing interpretability but also in minimizing the dimensionality of the feature variables. The suggested approach is tested on a real AD dataset of 2149 patients and 34 diverse clinical, demographic, and lifestyle characteristics. Experimental findings show that LSSSC outperforms traditional supervised and ensemble frameworks, with an accuracy of 95.35%, macro-F1 score of 0.9490, and AUC of 0.9513. Moreover, the combination of SHAP and LIME provides global and local interpretability, permitting a meaningful understanding of the model’s predictions. These contributions make LSSSC a strong, scalable, and interpretable framework for early AD prediction and promise much with respect to its application to real-world clinical decision-support systems.