Review on Malware Detection and Classification Using Machine Learning and Deep Learning Techniques

Abstract

Malicious software (malware) is increasing at an alarming rate, and modern malware routinely uses obfuscation techniques to evade detection, making reliable identification an open problem in cybersecurity. Signature- and heuristic-based approaches remain fast and effective against known malware but fail against unknown and zeroday variants, while behavior-based, model-checking-based, and cloud-based approaches perform better against unknown and complex threats; deep-learning-, mobile-, and IoT-based approaches have since emerged to address specific portions of the known and unknown malware landscape. This paper presents a structured literature review of malware detection approaches spanning signature-based, behavior-based, heuristic-based, model-checking-based, and deep-learning-based techniques, drawing on more than 110 primary studies published mainly between 2000 and 2020. For each approach, the underlying methodology, representative studies, reported detection performance, and documented limitations are summarized and compared. The review finds that no single approach reliably detects all malware in the wild: signature- and heuristic-based methods are fast but limited to known threats, behavior- and model-checking-based methods generalize better to unknown malware but carry higher false-positive rates and resource overhead, and deeplearning-based methods substantially reduce manual feature engineering yet remain vulnerable to adversarial evasion. These findings underscore both the difficulty of the underlying detection problem, which has been shown to be NP-complete, and the need for further research. The review concludes by motivating hybrid detection strategies that combine complementary techniques, such as behavior-based analysis with model checking or deep learning with cloud-based resources, as a promising direction for future malware detection research.