AI-Driven Integrated Data Science Framework for Intelligent Cyber Threat Detection and Adaptive Defense
TL;DR
This paper provides a unified framework to combine multiple-source data ingestion, automated feature engineering, ensemble-based detection, and reinforcement learning-based adaptive defense orchestration for AI to establish its suitability for deployment in real time for enterprise security operations.
Abstract
Cyber threats are evolving, becoming more complex, and require intelligent and adaptive security solutions that can integrate and function seamlessly on a diverse set of network types. In this paper, we provides a unified framework to combine multiple-source data ingestion, automated feature engineering, ensemble-based detection, and reinforcement learning-based adaptive defense orchestration for AI. The framework’s detection engine is a hybrid system that uses both gradient boosted decision trees and deep neural networks, along with graph-based anomaly detection, to ensure high accuracy with low computational complexity. Integrated SHAP based explain ability facilitates understanding of detection decisions by security analysts. The detection accuracy is 98.2% on NSL-KDD, CICIDS2017 and UNSW-NB15 datasets, which is superior to state-of-the-art detection methods by 4-7%. The framework was able to detect flows with an average latency of 8.3ms; establishing its suitability for deployment in real time for enterprise security operations.