Agentic AI Powered Autonomous Penetration Testing Framework with Real-Time Decision Making

Abstract

Adaptive and intelligent models are used to detect network intrusions and minimize the number of false alarms sent, based on the dynamic traffic pattern. In this project, an autonomous cyber threat assessment framework (Dueling DQN) using reinforcement learning principles is proposed for its design and development. A state vector is created for each network session, which contains data on packet size, protocol type, number of login attempts, length of login, encryption type, reputation score of the IP address, number of failed logins, browser type, and atypical use times. A custom reward mechanism is designed to incentivize attack detection with higher rewards and to have higher penalties for failing to detect attacks. To ensure the stability and generalizability of learning, the proposed model uses the Dueling DQN architecture, target network stabilization, prioritized experience replay, epsilon-greedy exploration, and also adds batch normalization, dropout regularization, and gradient clipping. The trained model classifies the traffic as an attack or benign traffic with a probability threshold optimized by the model. As a result of the experiment, on the dataset of cyber security intrusions with 9537 records, the accuracy value obtained is 87.53%, the precision value 89.12%, the recall value 87.53%, F1-value 87.23%, the ROC-AUC value 87.62%, and the average precision value 90.53%. It also allows for automated threat intelligence reports with the assistance of AI that provides attack chain analysis, risk interpretation, and remediation suggestions. The obtained results demonstrate the effectiveness and adaptability of Reinforcement Learning for intelligent intrusion detection and automated cybersecurity decision support.