Artificial Intelligence for Malware Detection in Software-Defined Networks: A Comprehensive Systematic Literature Review

Abstract

Software-Defined Networking has emerged as a fundamental networking paradigm for cloud computing, Internet of Things, fifth-generation (5G) communication, and data-center infrastructures because of its centralized control, programmability, and flexible network management. However, the logical centralization of the control plane also introduces significant security vulnerabilities, making SDN increasingly susceptible to malware, botnets, ransomware, Distributed Denial-of-Service, and other sophisticated cyberattacks. Artificial Intelligence (AI)-based malware detection techniques have gained considerable attention due to their capability to identify complex and previously unseen attack patterns. This paper presents a comprehensive Systematic Literature Review of intelligent malware detection approaches for SDN by following the PRISMA 2020 framework and Kitchenham guidelines. A total of 30 primary studies published between 2020 and 2026 were systematically selected, assessed, and synthesized. The reviewed literature was classified into three major categories: Machine Learning, Deep Learning, and Hybrid AI approaches, followed by comprehensive comparative analyses of datasets, learning algorithms, feature engineering strategies, evaluation metrics, detection performance, and reported limitations. The quantitative synthesis indicates a clear research transition from conventional ML techniques toward deep learning and hybrid intelligence frameworks, with hybrid models consistently shows the highest detection performance, frequently exceeding 99% detection accuracy. The analysis further reveals that Random Forest, Support Vector Machine, Convolutional Neural Networks, Long Short-Term Memory networks, Deep Neural Networks, and CNN–LSTM hybrid architectures are among the most widely adopted algorithms, whereas NSL-KDD, InSDN, UNSW-NB15, CICIDS2017, and IoT-23 remain the dominant evaluation datasets. The review identifies several persistent challenges, including dependence on benchmark datasets, limited real-world SDN validation, class imbalance, high computational complexity, insufficient explainability, limited cross-dataset generalization, and the absence of standardized benchmarking protocols. The review outlines future research directions emphasizing lightweight and explainable AI models, graph neural networks, federated and continual learning, adaptive hybrid intelligence, and standardized real-world evaluation frameworks to support the development of accurate, scalable, robust, and deployable malware detection systems for next-generation Software-Defined Networks.