Cyber Task Automation With Knowledge-Infused Reinforcement Learning and LLM-Guided Policies

Abstract

As cyber threats continue to evolve, there is a need for Autonomous Cyber Defense (ACD) strategies capable of fast and context-aware responses. Reinforcement learning (RL) has shown promise in automating cyber defense by exploring and learning effective countermeasures. However, RL often struggles with sparse reward signals and insufficient context to handle diverse attack scenarios. Furthermore, the convergence time of an RL agent is often high, making it difficult to train the agent in online settings. To address these challenges, we propose a large language model (LLM)-enhanced RL method that builds and queries a knowledge base (KB) derived from agent–environment interactions. We leverage the pre-trained knowledge of an LLM on different cybersecurity frameworks and use the LLM to analyze parts of the KB to generate appropriate actions for the RL agent. The LLM-generated output is infused into the RL training process to improve performance and reduce convergence time. To validate our approach, we formulate two RL problems: a contextual bandit problem, which accounts for possible misclassifications of network flows by the detection module, and a multi-step RL problem, which considers that adversarial actions may be missed by monitoring or detection tools. For the contextual bandit problem, we develop a custom environment guided by the MITRE ATT&CK framework, while for the multi-step RL problem, we use a prominent Cybersecurity simulation platform named CybORG. Experimental results show that our proposed approach outperforms the baseline RL by over 75% and 65% in the contextual bandit and multi-step RL settings, respectively, in terms of selecting more effective actions.