Reinforcement Learning Approach for Cybersecurity Threat Detection

Abstract

Traditional intrusion detection systems (IDS) struggle to detect evolving cyber threats due to their reliance on static signatures and fixed decision boundaries. Existing machine learning-based approaches partially address this limitation but often fail to generalize to zero-day attacks and lack adaptability in dynamic network environments. To address these challenges, this paper proposes a reinforcement learning-based intrusion detection system (RL-IDS) that model’s detection as a sequential decision-making problem using flow-level telemetry. The framework is implemented on the CIC-IDS2017 dataset with an isolated zero-day partition within a custom OpenAI Gym environment, incorporating asymmetric reward design, curriculum learning, entropy annealing, and early stopping to train Q-learning, Deep Q-Network (DQN), and Proximal Policy Optimization (PPO) agents. Experimental results show that the PPO-based RL-IDS achieves an F1-score of 0.857 with less than 4% false positives on known attacks, outperforming both DQN and a 400-tree Random Forest baseline. More importantly, it detects 27.7% of previously unseen zero-day attacks (Heartbleed and Infiltration), where the Random Forest fails completely. The system also processes over 290,000 flows per second, demonstrating real-time feasibility. These results demonstrate that reinforcement learning enables a practical balance between accuracy, adaptability, and efficiency, making it a promising solution for next-generation intrusion detection systems.