L-ARLPT: An LLM-Augmented Reinforcement Learning Framework for Autonomous Penetration Testing

Abstract

In recent years, Deep Reinforcement Learning (DRL) has emerged as a promising approach for automating penetration testing due to its capability to perform sequential decision-making in complex environments. However, in real-world enterprise networks, attack actions are typically characterized by highly coupled multi-dimensional parameter combinations, resulting in an exponentially expanding discrete action space. Such a large action space significantly degrades exploration efficiency and prevents conventional DRL agents from learning effective attack paths under sparse-reward conditions. To address these challenges, this paper proposes a Large Language Model-enhanced Autonomous Reinforcement Learning Penetration Testing framework (L-ARLPT). Specifically, the framework leverages the domain knowledge embedded in a Large Language Model (LLM) to perform tactical planning, thereby pruning the original action space into a compact set of candidate actions. Subsequently, an experience-driven layer employs the optimization mechanism of a Deep Q-Network (DQN) to conduct value estimation and policy learning within the reduced candidate set. To validate the effectiveness of the proposed framework, a high-fidelity enterprise penetration-testing simulation environment was constructed based on realistic enterprise attack scenarios. Experimental results demonstrate that, in a high-fidelity enterprise penetration-testing environment with a raw theoretical parameter-combination space containing 6×107 combinations, the proposed L-ARLPT framework achieves an average penetration depth of 3.33 out of 4.00, substantially outperforming both reinforcement learning baselines (all ≤1.27) and LLM-based baselines (1.12). Moreover, successful episodes require only 128.43 decision steps on average, enabling long-horizon, cross-domain lateral penetration in high-dimensional discrete action spaces.