Unsupervised deep learning for IoT botnet detection in a surveillance VLAN via multi-source traffic, threat intelligence and honeypot corroboration

Abstract

The increase in internet of thing devices, especially within VLANs in corporate networks, introduces significant security risks from advanced botnet attacks. Traditional signature-based detection methods struggle to identify encrypted, stealthy command-and-control traffic, while high false-positive rates overwhelm security teams with excess data. This study proposes an unsupervised deep-learning detection system using autoencoder (AE)-family models that learn normal traffic behaviors and identify anomalies through reconstruction error (MSE). An architecture comparison across five random seeds shows that a simple convolutional neural network (CNN)-AE performs similarly to a CNN-long short-term memory-AE (mean ROC-AUC difference of −0.0195 ± 0.0194); the hybrid is less stable and slower. The key contribution is the integrated, operational pipeline with a transparent evaluation approach, not architectural innovation. Tested on multi-source metadata—Wireshark, firewall logs, T-Pot honeypot—collected over fifteen days from a real enterprise VLAN, the system incorporates external threat intelligence (AbuseIPDB, OTX) and contextual behaviors through a multi-dimensional scoring system (MDSS), converting raw detections into prioritized risk scores. Under deployment, the model flags 3.75% of traffic as anomalous, with approximately 1% false positives on benign devices. Stress tests—including feature-leakage ablation, multiple seed runs, temporal holdouts, and clean-device testing—suggest a realistic record-level ROC-AUC of 0.72–0.78, compared to near-perfect (>0.99) results in saturated IP-partition testing, which is reported solely as an upper bound. The MDSS assigns risk tiers: 0 Very-High, 2 High, 26 Medium, and 37 Low for the 65 monitored devices. Anomalies are independently validated via two MITRE ATT&CK pipelines—one focused on honeypot/policy sources and one based solely on the model’s detections—both identifying the same techniques (T1071, T1573, T1046, T1090). This confirms that model decisions are not circularly based on the evaluation labels. With low-latency, GDPR/KVKK-compliant, metadata-only analysis and operational prioritization, this framework bridges the gap between deep-learning security solutions’ theoretical potential and their real-world enterprise application. It provides a transparent, reproducible methodology for proactive, scalable, and interpretable botnet detection.