AJAD-BoostNet: A Hybrid Deep K-Means and XGBoost Framework for Automated Cancer Histopathological Image Classification

Abstract

Cervical and lung cancers remain among the leading causes of cancer-related mortality worldwide. This highlights the need for accurate, scalable, and computationally efficient computer-aided diagnostic systems. Deep learning has shown remarkable success in histopathological image analysis. However, most existing approaches depend on large annotated datasets, extensive computational resources, and end-to-end supervised training. To address these limitations, this study proposes AJAD-BoostNet (Adaptive Joint Analysis and Deep Boosting Network). This novel hybrid framework integrates deep transfer learning, unsupervised clustering, dimensionality reduction, and boosting-based refinement for automated cancer histopathological image classification. The proposed framework uses a pretrained ResNet50 model to extract high-level semantic representations from histopathological images. Then, Principal Component Analysis (PCA) reduces feature dimensionality while preserving discriminative information. The optimized feature space is clustered with the K-Means algorithm for initial unsupervised classification. To further enhance predictive performance and correct clustering errors, an XGBoost-based boosting module is applied as a misclassification refinement strategy. The framework was evaluated on two benchmark datasets: the IARC Cervical Cancer Image Bank and the LC25000-based Lung Cancer Histopathological Images dataset. Experimental results demonstrate the effectiveness of the proposed approach across both datasets. For lung cancer classification, AJAD-BoostNet achieved an accuracy of 97.08%, an F1-score of 97.00%, a Cohen's Kappa score of 94.17%, and an AUC of 97.08%. For cervical cancer classification, the framework attained an accuracy of 64.84%, an F1-score of 65.00%, a Cohen's Kappa score of 29.20%, and an AUC of 64.55%. These results surpass those of conventional CNN-based models trained under similar conditions. Additionally, 10-fold cross-validation and confidence interval analyses confirmed the robustness, stability, and generalization capability of the proposed framework. The findings indicate that combining deep semantic feature extraction with clustering and boosting provides an efficient and interpretable alternative to fully supervised deep learning architectures. This makes AJAD-BoostNet a promising solution for intelligent cancer diagnosis in resource-constrained clinical environments.