Explainable Machine Learning for Software Defect Prediction Using Random Forest and XGBoost
Abstract
Software defect prediction plays a critical role in quality assurance by identifying fault-prone modules prior to system deployment. Conventional tree-based machine learning approaches often operate as black boxes, making it difficult for software engineering teams to understand the key code attributes driving predictions. This study evaluates an explainable defect prediction framework that compares Random Forest and XGBoost classifiers on static code metrics while evaluating model transparency through Explainable Artificial Intelligence. Models were trained and optimized using Python, scikit-learn, and XGBoost on the NASA MDP KC1 dataset, comprising 2109 C++ software modules with 21 static metrics under a 5.47 to 1 class imbalance. Hyperparameter optimization was conducted using RandomizedSearchCV with 5-fold stratified cross-validation on an 80 to 20 stratified train-test split. On the test set, Random Forest achieved an accuracy of 0.7037, precision of 0.4151, recall of 0.3492, F1-score of 0.3793, and ROC-AUC of 0.6089. XGBoost achieved an accuracy of 0.6173, precision of 0.3295, recall of 0.4603, F1-score of 0.3841, and ROC-AUC of 0.6026. McNemar testing revealed significant prediction disagreement between models with a chi-squared value of 8.8889 and p-value of 0.0029, while tree-based SHAP analysis demonstrated moderate global rank agreement with a Spearman correlation coefficient of 0.570. Random Forest prioritized Halstead Volume whereas XGBoost prioritized Halstead Difficulty. Software engineering practitioners can integrate these explainable ensemble workflows into continuous integration pipelines to prioritize high-risk code modules for targeted refactoring and code review.