Jul 2026· Digital Threats: Research and Practice· 0 citations· 41 references
TL;DR
The findings in this study highlight the potential and current limitations of LLMs for phishing detection in dynamic instant messaging environments and emphasize the superior performance of platform-tailored models.
Abstract
Social media platforms have become valuable sources for detecting phishing due to their widespread use and rapid content sharing. Among them, Telegram stands out as an open, data-rich environment, offering access to large public groups where phishing links frequently appear. This study investigates whether Large Language Models (LLMs) can serve as effective tools for phishing detection on Telegram. We systematically evaluate three representative LLMs — GPT-4o, LLaMA-3.1, and DeepSeek V3 — across multiple prompting strategies and input configurations. For comparison, we employ an improved version of traditional feature-based machine learning models from the literature, enhanced with resampling techniques to address the strong class imbalance in our data. Experimental results show that while LLMs demonstrate some potential, achieving a Recall for the phishing class of up to 0.554 with DeepSeek — representing the model’s ability to correctly identify 55.4% of all actual phishing messages — they suffer from extremely low precision (0.065 with DeepSeek), resulting in a low F1-score of 0.114. In contrast, our feature-based models achieve a significantly more reliable precision-recall balance, with an overall phishing F1-score of 0.749 and a precision of 0.738. The findings in this study highlight the potential and current limitations of LLMs for phishing detection in dynamic instant messaging environments and emphasize the superior performance of platform-tailored models.
This work provides a multi-dimensional framework for evaluating phishing detection systems and demonstrates that for most real-world applications, a multilingual, efficient encoder like XLM-RoBERTa represents the most viable and trustworthy choice.
Armand Florent Tsafack Piugie, Mathieu Valois, Emmanuel Giguet et al.· International Conference on...· 0 citations
Experimental results on English and Korean SMS datasets indicate that LLMs are highly robust for spam detection and can provide interpretable rationales by leveraging their language understanding capabilities, suggesting that LLMs are a highly effective and practical option for modern SMS spam filtering.
Minseok Hur, Sooyon Seo, Jaeho Hwang et al.· International Journal of Inf...· 0 citations
Despite the advancements made by researchers, spam emails remain one of the biggest challenges in the field of cybersecurity. Spam emails can serve as phishing emails or carry viruses that compromise the security of an organization's system. Current detection techniques depend on supervised learning or rely on cloud-based services, which can compromise user data privacy and affect implementation flexibility. This paper evaluates the capability of five large language models (LLMs) in zero-shot spam email classification. The models used in this study include llama3.1:8b, deepseek-r1:8b, gemma3:4b, falcon3:7b, and mistral:7b. In addition to predicting whether the email is spam or not, the LLM was also asked to generate an explanation of its prediction in natural language form. The experiments were conducted on two benchmark datasets: the Ling and TREC2007 datasets. In terms of performance, llama3.1:8b outperformed other LLMs when evaluated on the TREC2007 dataset (98.78% accuracy) and deepseek-r1:8b had the best performance on the Ling dataset (98.79%). The results show that open-weight LLMs can achieve competitive spam detection performance in a local, privacy-preserving environment without any fine-tuning.
Vusal Shahbazov· 2026 7th International Confe...· 0 citations
Efficiency makes the proposed approach exceptionally suitable for real-time detection in resource-constrained environments, such as mobile applications or browser extensions, providing an accessible and proactive layer of defense for end-users.
Phishing remains one of the most persistent cyber threats, particularly in email environments where deceptive messages can be distributed at scale. This paper compares five classifiers: Multinomial Naive Bayes, Random Forest, Bidirectional Long Short-Term Memory (BiLSTM), DistilBERT, and BERT-base. A multi-source corpus of 82,689 cleaned and deduplicated emails was built from nine public datasets. Under a unified protocol, BERT-base achieved the highest F1-score (0.9824), while DistilBERT obtained an almost identical F1-score (0.9822) with lower measured inference latency (1.130 versus 2.218 ms/email), representing the strongest accuracy–latency trade-off in the evaluated environment. LIME explanations exposed plausible phishing indicators, such as urgency and account-verification language, but also mixed local contributions that require cautious interpretation. In the source-held-out experiment, the positive-class prevalence changed from 41.1% in training to 26.6% in testing, and DistilBERT produced 505 false negatives but only two false positives. Consequently, recall decreased from 0.9750 to 0.5346, showing that high mixed-source test performance does not guarantee robustness when complete data sources are unseen.
Andre Sebastian Samaniego Buñay, Ariel Misael Orellana Albarracin, Joel Marcelo Chuquimarca Pomagualli· Enfoque UTE· 0 citations
External validation against ten official Iraqi university websites resulted in correct classification with no false positives, demonstrating that the proposed phishing detection model is reliable.
