Optimizing Multiclass Android Malware Family Classification Using SMOTE-Tomek Links and XGBoost

Abstract

The increasing sophistication of Android malware attacks has created significant challenges for accurate malware family classification, particularly under highly imbalanced data distributions where minority malware families are frequently misclassified. This study presents a robust multiclass Android malware family classification framework by combining SMOTE-Tomek Links hybrid resampling with an optimized Extreme Gradient Boosting (XGBoost) classifier. The proposed framework addresses two critical issues in previous studies: ineffective handling of minority classes and potential data leakage during resampling and model validation. Experiments were conducted using the CCCS-CIC-AndMal-2020 After Reboot dataset containing 25,059 malware samples distributed across 14 malware families. The proposed approach applies stratified data partitioning, leakage-free SMOTE-Tomek Links integration within an imbalanced-learn pipeline, and RandomizedSearchCV-based hyperparameter optimization with 5-fold stratified cross-validation. Evaluation on an independent holdout test set demonstrates that the optimized framework achieves 80.09% accuracy, 79.85% weighted F1-score, 74.00% macro F1-score, and 97.48% OvR ROC-AUC, outperforming baseline XGBoost and Random Forest models. The results confirm that hybrid resampling combined with optimized gradient boosting improves classification reliability, especially in addressing severe class imbalance and enhancing recognition capability across diverse Android malware families.