A hybrid Fuzzy-Autoencoder based approach for resilient zero-day malware detection under adversarial conditions

Abstract

Amid the growing use of artificial intelligence (AI) and machine learning, the risk of adversarial zero-day malware attacks has increased and thus, enhancements to intelligent detection systems are necessary. Although one-class (OC) classifier-based detectors, particularly autoencoders, are widely adopted for zero-day malware detection, their robustness under adversarial conditions has not been fully explored. This work analyzes the robustness of OC malware detection approaches and demonstrates that conventional autoencoder (AE)-based detectors are highly dependent on rigid reconstruction-error thresholds, making them vulnerable to adversarial attacks. To overcome these limitations, a Fuzzy-Autoencoder (Fuzzy-AE) based approach is proposed, where the fuzzy logic replaces hard reconstruction-error thresholds and the deviation-driven fuzzy reasoning effectively detects adversarial attacks. To evaluate adversarial robustness, adversarial samples are generated using a generative adversarial networks (GANs) whose generator is trained to closely mimic legitimate portable executable samples. Experimental results demonstrate that the proposed Fuzzy-Autoencoder achieves a safety-aware accuracy of 97.08%, successfully prevents 99.97% of real malware samples and ensures that no adversarial sample is classified as legitimate. Furthermore, the obtained results highlight the effectiveness of the proposed approach for resilient and practical zero-day malware detection.