Fine-Tuning LLMs for Operational Phishing Email Detection
TL;DR
This work provides a multi-dimensional framework for evaluating phishing detection systems and demonstrates that for most real-world applications, a multilingual, efficient encoder like XLM-RoBERTa represents the most viable and trustworthy choice.
Abstract
: Phishing email attacks, a predominant and evolving cyber threat, necessitate detection systems that are not only accurate but also robust, efficient, trustworthy, and explainable. While Large Language Models (LLMs) offer transformative potential for semantic analysis in many areas, their practical deployment for phishing detection requires a holistic assessment beyond peak accuracy. This paper presents an operational phishing detection system that processes raw emails using a LLM and provides understandable explanations. We compared three fine-tuned LLMs among CamemBERT-base, XLM-RoBERTa, and Mistral 7B and we evaluate their performance, generalization capabilities, computational efficiency, and explainability on multiple significant datasets. Our results demonstrate that LLMs significantly outperform character-level models, with XLM-RoBERTa and Mistral 7B achieving over 99.4% accuracy. Critically, we reveal a fundamental trade-off: while Mistral 7B achieves marginally higher accuracy, its computational cost could be prohibitive for real-time deployment. In contrast, XLM-RoBERTa provides an optimal balance, offering state-of-the-art accuracy (99.46%), superior cross-dataset generalization (95.76% on an unseen corpus), and practical inference speed. Furthermore, by integrating SHAP-based explainability with a generative LLM, we validate that model decisions align with recognizable phishing indicators, bridging the gap between high performance and operational trust. This work provides a multi-dimensional framework for evaluating phishing detection systems and demonstrates that for most real-world applications, a multilingual, efficient encoder like XLM-RoBERTa represents the most viable and trustworthy choice.