SHAP-PDFStack: An Explainable Hybrid Stacking Model with Latent Feature Learning for Malicious PDF Detection

Abstract

Malicious PDF documents have become one of the common attack vectors in modern cybersecurity because they can contain embedded JavaScript code, malicious URLs, and obfuscated content. Although many machine learning, deep learning, and explainable artificial intelligence methods have been proposed for malicious PDF detection, most existing approaches depend on handcrafted features, do not use latent feature learning, employ simple architectures, or provide limited model interpretability. This paper proposes SHAP-PDFStack, an explainable hybrid model that combines latent feature learning, ensemble learning, and adaptive explainability. The proposed model includes four phases. First, data preprocessing is applied to clean and normalize the dataset. Second, an autoencoder extracts compact latent features from the original data. Third, Random Forest, XGBoost, LightGBM, and CatBoost are combined through an Out-of-Fold stacking strategy. Finally, SHAP-based adaptive weighting is applied to improve prediction transparency and classifier fusion. Experiments were conducted on the CIC-Trap4Phish 2025 dataset, which contains 19,296 samples and 40 input features. The proposed model achieved 99.8% accuracy, 99.9% precision, 99.9% recall, and 99.9% F1-score, and it outperformed conventional machine learning and ensemble learning methods. In addition, SHAP analysis explained the contribution of individual classifiers and increased the reliability of the model, which makes it suitable for practical applications.