A hybrid machine learning framework for dynamic malware classification using CNN, LSTM, and Random Forest models
Malware continues to evolve through polymorphic, metamorphic, and file-less techniques, reducing the effectiveness of signature-based detection. This study proposes a hybrid dynamic malware classification framework that combines Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) networks, and a Random Forest (RF) classifier. Dynamic behavioural traces are transformed into machine-readable features using Term Frequency-Inverse Document Frequency (TF-IDF), Count Vectorization, and Word2Vec embeddings. CNN and LSTM models are used as feature extractors to capture local and sequential behavioural patterns, while RF performs the final classification to improve robustness and interpretability. The reported experimental evaluation shows that the embedding-based hybrid CNN-RF configuration achieved the best performance, reaching 98.21% accuracy. The findings indicate that combining deep representation learning with ensemble classification can improve dynamic malware detection and reduce dependence on static signatures. The proposed framework is therefore a promising approach for adaptive malware classification and future security monitoring applications.