Failure-Based Testing for Deep Reinforcement Learning Agents

Evaluating Fuzz Testing for Reinforcement Learning Agents

Reinforcement Learning (RL) agents are increasingly deployed in safety-critical domains such as robotics, autonomous driving, and drone control, where unexpected behaviors may lead to severe real-world consequences. Fuzz testing has recently emerged as a promising method for exploring the vast state spaces of RL agents and exposing crashes. Although numerous RL fuzzing methods have been proposed, existing studies often differ in evaluation settings, baselines, and metrics, making it difficult to draw reliable conclusions about their relative effectiveness and practical usefulness. To address this gap, we present the first comprehensive empirical study that systematically evaluates RL fuzzing methods from four complementary perspectives: effectiveness, diversity, efficiency, and practical utility. We benchmark five state-of-the-art methods alongside random testing under unified configurations across three environments of increasing complexity (MountainCar, BipedalWalker, and CARLA), and further assess the downstream usefulness of detected crashes for agent robustness improvement and safety monitoring. Our results reveal several key insights. For instance,throughput-oriented methods like MDPFuzz demonstrate superior effectiveness and efficiency in crash discovery, while methods explicitly designed to encourage exploration like SeqDivFuzz excel at uncovering diverse crash behaviors. We also show that fuzzing-generated crashes can meaningfully improve agent robustness and enable accurate safety monitoring with strong cross-method generalization. Beyond these empirical findings, we distill actionable guidance for both researchers and practitioners, highlighting the benefits of combining complementary fuzzing strategies and adopting multi-level diversity analysis to achieve more comprehensive and practical RL testing.

Symbolic Knowledge Transfer for Sample-Efficient Deep Reinforcement Learning

Reinforcement Learning (RL) provides a principled framework for sequential decision-making in complex environments. However, state-of-the-art Deep Reinforcement Learning (DRL) algorithms typically require large amounts of training data and often fail to generalize beyond small-scale training scenarios, even on standard benchmarks. We propose a neuro-symbolic DRL approach that incorporates background symbolic knowledge to improve both sample efficiency and generalization to more challenging, unseen tasks. Specifically, partial policies learned in simple domain instances, where high performance can be achieved reliably, are transferred as structured priors to accelerate learning in more complex environments, eliminating the need to tune DRL parameters from scratch. Our method represents partial policies as logical rules in the Answer Set Programming (ASP) formalism and performs online reasoning to guide training through two complementary mechanisms: (i) biasing the action distribution during exploration, and (ii) rescaling Q-values during exploitation. This integration of ASP reasoning with DRL enhances interpretability and trustworthiness while accelerating convergence, particularly in sparse-reward settings and tasks with long planning horizons, without introducing significant computational overhead. We empirically evaluate our approach on challenging variants of gridworld environments under both fully and partially observable settings. Results demonstrate consistent performance improvements over a state-of-the-art reward machine baseline.

L-ARLPT: An LLM-Augmented Reinforcement Learning Framework for Autonomous Penetration Testing

In recent years, Deep Reinforcement Learning (DRL) has emerged as a promising approach for automating penetration testing due to its capability to perform sequential decision-making in complex environments. However, in real-world enterprise networks, attack actions are typically characterized by highly coupled multi-dimensional parameter combinations, resulting in an exponentially expanding discrete action space. Such a large action space significantly degrades exploration efficiency and prevents conventional DRL agents from learning effective attack paths under sparse-reward conditions. To address these challenges, this paper proposes a Large Language Model-enhanced Autonomous Reinforcement Learning Penetration Testing framework (L-ARLPT). Specifically, the framework leverages the domain knowledge embedded in a Large Language Model (LLM) to perform tactical planning, thereby pruning the original action space into a compact set of candidate actions. Subsequently, an experience-driven layer employs the optimization mechanism of a Deep Q-Network (DQN) to conduct value estimation and policy learning within the reduced candidate set. To validate the effectiveness of the proposed framework, a high-fidelity enterprise penetration-testing simulation environment was constructed based on realistic enterprise attack scenarios. Experimental results demonstrate that, in a high-fidelity enterprise penetration-testing environment with a raw theoretical parameter-combination space containing 6×107 combinations, the proposed L-ARLPT framework achieves an average penetration depth of 3.33 out of 4.00, substantially outperforming both reinforcement learning baselines (all ≤1.27) and LLM-based baselines (1.12). Moreover, successful episodes require only 128.43 decision steps on average, enabling long-horizon, cross-domain lateral penetration in high-dimensional discrete action spaces.

Generalization in Reinforcement Learning from Logical Specifications

Advances in Multi-Agent Deep Reinforcement Learning: Methods with Applications and Challenges

Multi-agent deep reinforcement learning (MARL) extends deep reinforcement learning (DRL) to environments involving multiple interacting agents and has enabled applications in domains such as autonomous vehicles, robotics, unmanned aerial vehicles (UAVs), and multi-player games. Compared with single-agent learning, MARL introduces additional challenges, including non-stationarity, partial observability, multi-agent credit assignment, and scalability. This paper presents a narrative survey of recent developments in MARL and discusses major approaches proposed to address these challenges. In particular, we examine research directions centred on centralised training with decentralised execution (CTDE), value decomposition, learned communication, graph-based methods, and model-based learning. We further discuss commonly used benchmark environments and evaluation practices, highlighting considerations related to reproducibility, robustness, and generalisation. Finally, we outline open research challenges and future directions concerning theoretical understanding, sample efficiency, scalable coordination, and deployment in real-world settings. Rather than providing an exhaustive systematic review, this survey aims to offer an organised and up-to-date synthesis of recent progress in MARL.

Explainable Reinforcement Learning via Physics-Aware Policy Distillation

In safety-critical sectors such as robotics and automotive engineering, the deployment of Deep Reinforcement Learning (DRL) is often hindered by the black-box nature of deep neural networks. This lack of transparency poses significant challenges for regulatory compliance and human-agent trust. This paper presents an experimental study aimed at making high-performance continuous control DRL systems interpretable. A policy distillation framework is implemented using the classic Inverted Pendulum benchmark. A high-performance Twin Delayed DDPG (TD3) agent serves as an opaque, continuous teacher model, whose policy is distilled into an interpretable student surrogate based on a shallow Decision Tree. By leveraging a custom physics-aware feature and"Noisy Oracle Rollouts"for dataset generation, the distillation process achieves performance equivalent to the expert teacher. Furthermore, comparative control theory analysis reveals a fundamental trade-off: transitioning from continuous to discrete rule-based control induces high-frequency Bang-Bang actuation and a stable bimodal limit cycle. Simulation results indicate that Bounded-Input Bounded-Output (BIBO) stability is maintained while providing both global and local interpretability for safe autonomous systems.