Explainable Gradient Boosting for Transparent Phishing URL Detection: A Cross-Dataset, Statistically Validated SHAP Analysis of XGBoost, LightGBM, and CatBoost

Abstract

: Phishing remains one of the most persistent attack vectors in cybersecurity, and the gradient-boosting models that now dominate its automated detection are frequently deployed as opaque classifiers, which limits analyst trust and slows incident response. This paper presents a cross-dataset, explainability-driven evaluation of three gradient boosting algorithms–XGBoost, LightGBM, and CatBoost—for phishing URL classification, paired with a SHAP (Shap-ley Additive Explanations)-based framework that attributes every prediction to specific, human-readable features rather than treating the classifier as a black box. To eliminate a data leakage risk present in an earlier evaluation protocol—in which hyperparameter tuning and final evaluation shared the same held-out split—all experiments reported here use a corrected 60:20:20 train/validation/test protocol, with a validation-only random hyperparameter search and a test partition touched exactly once per dataset. Under this protocol, on the original 2015–2017 benchmark (9581 samples, 48 features), XGBoost reached 99.11% test accuracy (F1 = 0.991, ROC-AUC [Receiver Operating Characteristic–Area Under the Curve] = 0.999), with LightGBM and CatBoost each at 98.70% (F1 = 0.986); paired McNemar tests and five-seed repeated runs show these differences are not statistically significant ( p ≥ 0.077); this significance testing was performed on this benchmark only and was not repeated on the other three datasets. The same pipeline was replicated on three additional, more recent phishing datasets (PhiUSIIL, 2023; a 2026 URL-Phish benchmark; and LegitPhish, 2025), where accuracy ranged from 96.08%–96.27% on the hardest, most recent benchmark to near-ceiling performance on PhiUSIIL—a result we treat with caution given the disproportionate influence of a single similarity-index feature. A quantitative, reproducible criterion for SHAP feature reliance (mean ∣ SHAP ∣ exceeding 5% of a model’s per-dataset maximum) shows that all three models rely on a substantially narrower evidence base than earlier, leakage-affected estimates suggested–roughly half of the 48 features on the original benchmark. We further test, rather than merely hypothesize, whether narrower SHAP reliance predicts adversarial vulnerability: a SHAP-guided perturbation experiment across all four datasets shows that XGBoost, not the narrowest-relying LightGBM, is the most susceptible to feature-guided evasion (50.0% mean evasion rate vs. 32.4% for LightGBM), indicating that SHAP breadth alone does not predict robustness. Training and inference time, per-row SHAP computation cost, classification-threshold sensitivity, probability calibration, and performance under a realistic 90:10 class imbalance are also reported. These findings support feature-attribution analysis as a useful complement to accuracy when comparing similarly performing gradient boosting models, while showing that some of the more intuitive claims about that attribution require direct testing rather than inference.