Behavior-Based Ransomware Detection Through File System Change Analysis Using Random Forest in a Virtual Environment

Abstract

Ransomware has become one of the most significant cybersecurity threats due to its ability to encrypt files and disrupt system availability. Conventional signature-based detection methods often struggle to identify newly emerging ransomware variants, making behavior-based approaches a promising alternative. This study proposes a behavior-based ransomware detection method based on file system change analysis using the Random Forest algorithm in a virtual environment. File activities, including creation, modification, deletion, renaming, and extension changes, were monitored and transformed into behavioral features for classification. A total of 6,400 samples were generated and divided into training and testing datasets using an 80:20 ratio. The Random Forest model was trained to distinguish normal file activities from ransomware behavior based on extracted behavioral features. Experimental results showed that the proposed model achieved 100% accuracy, precision, recall, and F1-score on the testing dataset. To further evaluate the robustness of the model, additional manual scenario testing was conducted using representative normal activities (backup ZIP and cloud synchronization) and ransomware behaviors (slow ransomware and super slow ransomware). The proposed model consistently classified all representative scenarios correctly, demonstrating its ability to distinguish legitimate high-volume file operations from ransomware activities and to detect ransomware even when encryption was performed gradually. These results indicate that the proposed behavior-based approach is effective for early ransomware detection and has the potential to support real-time ransomware detection systems in practical environments.