A controlled, pairwise comparison of two phishing-detection approaches shows that clean-data accuracy does not predict adversarial robustness, and that adversarial testing should be a standard part of phishing-detection evaluation.
Abstract
Phishing emails remain one of the most persistent cybersecurity threats, and machine-learning classifiers are widely used to detect them. Most reported detection accuracies, however, are measured on clean, in-distribution test data rather than on emails deliberately altered to evade detection. This paper reports a controlled, pairwise comparison of two phishing-detection approaches a TF-IDF + Logistic Regression baseline and a fine-tuned DistilBERT transformer trained on a unified corpus of 82,255 emails drawn from six public datasets and evaluated under three conditions: normal in-distribution, synthetic phishing, and adversarial phishing. Both models exceeded 98% accuracy on clean data yet degraded sharply under adversarial testing: TF-IDF + LR fell to 64.00% (a 34.59-percentage-point drop) and DistilBERT fell to 63.64% (a 35.40-percentage-point drop) a gap of only 0.36 percentage points, equivalent to a single email in the 275-sample adversarial test set. LIME, SHAP, and attention-rollout analysis indicate the two models relied on different evidence yet showed similar vulnerability. Pairwise error analysis shows the models agreed on 54.9% of adversarial samples but each made a similar number of exclusive errors (24 and 25 respectively), indicating partly complementary rather than identical failure modes. The results show that clean-data accuracy does not predict adversarial robustness, and that adversarial testing should be a standard part of phishing-detection evaluation.
PhishingGAT, a detector that fuses word-level semantic features with structural ones and is hardened against adversarial perturbation, is presented, a detector that fuses word-level semantic features with structural ones and is hardened against adversarial perturbation.
R. Kodali, Siva Rama Krishna T Dr· International Journal of Inn...· 0 citations
Phishing remains one of the most persistent cyber threats, particularly in email environments where deceptive messages can be distributed at scale. This paper compares five classifiers: Multinomial Naive Bayes, Random Forest, Bidirectional Long Short-Term Memory (BiLSTM), DistilBERT, and BERT-base. A multi-source corpus of 82,689 cleaned and deduplicated emails was built from nine public datasets. Under a unified protocol, BERT-base achieved the highest F1-score (0.9824), while DistilBERT obtained an almost identical F1-score (0.9822) with lower measured inference latency (1.130 versus 2.218 ms/email), representing the strongest accuracy–latency trade-off in the evaluated environment. LIME explanations exposed plausible phishing indicators, such as urgency and account-verification language, but also mixed local contributions that require cautious interpretation. In the source-held-out experiment, the positive-class prevalence changed from 41.1% in training to 26.6% in testing, and DistilBERT produced 505 false negatives but only two false positives. Consequently, recall decreased from 0.9750 to 0.5346, showing that high mixed-source test performance does not guarantee robustness when complete data sources are unseen.
Andre Sebastian Samaniego Buñay, Ariel Misael Orellana Albarracin, Joel Marcelo Chuquimarca Pomagualli· Enfoque UTE· 0 citations
Machine learning has become a widely adopted approach for URL-based phishing detection, with many studies reporting F1 scores exceeding 0.95 on benchmark datasets. However, recent adversarial machine learning research has questioned the robustness of these models, suggesting that small input perturbations can severely degrade detection performance. In this study, we argue that a substantial part of this reported vulnerability stems from the way adversarial attacks are evaluated. Specifically, many existing studies assess attacks in the feature space, where feature values are modified directly without ensuring that the resulting samples correspond to valid, functional URLs. To investigate this issue, we conduct a two-stage empirical study using both a benchmark feature dataset and a dataset of real phishing URLs. Crucially, to avoid confounding the attack space with dataset differences, we additionally evaluate both feature-space and problem-space attacks on the same real-URL dataset, using an identical model and manipulable-feature budget. Our experiments reveal a striking contrast between these evaluation settings. While feature-space attacks reduce the detection rate of a Random Forest classifier on the benchmark dataset from 0.96 to 0.36, analogous manipulations performed on real URLs have almost no effect on detection performance, as the most informative signals originate from host-related attributes that are difficult for attackers to manipulate. Building on this observation, we propose a set of robust features that capture stable domain characteristics, including lexical word validity, homoglyph disguises, brand impersonation, subdomain depth, character entropy, and transport-related signals. Incorporating these features substantially improves robustness under adversarial conditions, maintaining phishing detection rates between 0.24 and 0.76 where the lexical-only baseline deteriorates to zero under a non-adaptive attacker, while also increasing the clean-data F1 score from 0.985 to 0.994. We further evaluate an adaptive attacker that explicitly targets the proposed features; although the proposed representation raises the attacker’s cost and helps under moderate attacks, host-derived features remain the only strictly attack-invariant component, so we position the proposed features as a complement to host-based signals rather than a standalone defense. Additional analyses, including model comparison, hyperparameter sensitivity analysis, feature ablation, SHAP-based interpretation, multi-seed confidence intervals, a domain-disjoint evaluation, and host-only evaluation, consistently support the proposed approach. The findings demonstrate that problem-space evaluation provides a more realistic assessment of adversarial robustness than conventional feature-space testing and show that robust feature engineering offers a practical strategy for developing phishing detection systems that remain effective under realistic adversarial conditions.
It is demonstrated that clean-text performance is not a reliable predictor of adversarial robustness, and the results underscore the necessity for architecture-specific defences and frame smishing detection as an adversarial cybersecurity challenge rather than a static classification task.
Denzel Chiuseni, A. Bahizire, Silva Hama et al.· 0 citations
—Proposed cybercrime detection models demonstrate satisfactory performance on specific benchmark datasets; however, they are not always robust across diverse, practical scenarios. This paper examines the generalization performance of a cross-dataset measure for cybercrime detection across network traffic, phishing email, and malicious Uniform Resource Locator (URL) domains. The work combines three popular security benchmarks Canadian Institute for Cybersecurity Intrusion Detection System Dataset2017 (CIC-IDS2017), Canadian Institute for Cybersecurity Phishing Dataset2019 (CIC-Phishing2019), and Malicious URL 2020 in a single preprocessing and learning pipeline to avoid dataset-specific bias. One or more datasets are used to train models, which are then directly evaluated on previously unseen datasets to verify transferability across distribution shifts. We will discuss performance while considering accuracy, F1 − Score, Receiver Operating Characteristic (ROC), and fold-to-fold stability. Experiments demonstrate that direct training with a single random source results in significant performance deterioration, with a 23% decrease in F1 − Score when applied to unseen datasets. In contrast, the degradation caused by the proposed framework is kept below 10% and maintains Receiver Operating Characteristic–Area Under the Curve (ROC–AUC) values consistently above 0.90. Paired significance testing demonstrates that the gains in robustness are highly significant ( p < 0.01). The results indicate that cross-dataset evaluation is essential for the
R. Adinarayana, G. V. Krishna· Journal of Advances in Infor...· 0 citations
Experimental evaluations on the UCI SMS Spam Collection dataset demonstrate that the TF-IDF-PCA-DT pipeline achieves a detection accuracy of 99% while reducing model size by 77% and inference time by 75%, demonstrating the practicality of the proposed framework for real-world deployment in resource-constrained environments.
Baraa Tareq Hammad, Ismail Taha Ahmed, Mohamed A. Hafez et al.· De Computis· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
