Explainable ensemble learning framework with recursive feature elimination for lung cancer stage prediction using demographic and lifestyle parameters

Abstract

Lung cancer remains one of the leading causes of cancer-related mortality worldwide, primarily due to challenges in early-stage detection and accurate risk stratification. Conventional machine learning models often exhibit limited interpretability and reduced predictive capability when modelling complex, non-linear relationships among demographic and lifestyle risk factors. Therefore, robust, explainable, and methodologically rigorous predictive frameworks are required to support reliable clinical decision-making. This study proposes a systematically validated explainable ensemble learning framework for lung cancer stage prediction by integrating optimized stacking and voting strategies. A leakage-free machine learning pipeline comprising data preprocessing, normalization, recursive feature elimination (RFE), and GridSearchCV-based hyperparameter optimization was developed. Multiple machine learning classifiers were combined through optimized ensemble learning, while SHapley Additive exPlanations (SHAP) and Local Interpretable Model-agnostic Explanations (LIME) were employed to provide complementary global and local interpretability. Model performance was evaluated using an independent hold-out test set, stratified 10-fold cross-validation, 95% confidence intervals, non-parametric statistical analysis using the Friedman aligned-ranks test with Holm post-hoc analysis, and a component-wise ablation study. The optimized stacking model (STB) achieved the best overall predictive performance, with an accuracy of 99.52%, precision of 99.57%, recall of 99.50%, F1-score of 99.50%, and AUC of 100%. Statistical validation confirmed the robustness and comparative performance of the proposed framework, while the ablation study verified the contribution of feature selection, ensemble learning, and hyperparameter optimization. SHAP and LIME consistently identified passive smoking (PS), obesity (OB), coughing of blood (CB), wheezing (WH), and fatigue (FT) as the most influential predictors of lung cancer stage. The proposed explainable ensemble learning framework provides accurate, robust, and interpretable lung cancer stage prediction through the integration of leakage-free model development, optimized ensemble learning, explainable artificial intelligence, statistical validation, and component-wise ablation analysis. These findings demonstrate its potential as a reliable decision-support framework while highlighting the need for future validation using independent multi-center clinical datasets.