Skip to content
Open access

Network Traffic Anomaly Detection for Server Security Using a Hybrid CNN–BiLSTM and LightGBM Optimized by Whale Optimization Algorithm

Aug 2026 · International Journal of Advances in Data and Information Systems · Vol 7, pp. 656-670 · 0 citations · 25 references

TL;DR

A hybrid anomaly detection pipeline that combines deep learning for representation learning with gradient-boosted decision trees for multiclass classification on structured traffic features is proposed, designed with practical deployment in mind by leveraging a tree-based classifier on learned features.

Abstract

The rapid growth of server-based digital services (e.g., web services, databases, cloud platforms, and IoT gateways) expands the attack surface, while threats such as DDoS, brute-force attempts, port scanning, web attacks, bot activity, and infiltration are increasingly difficult to distinguish from legitimate traffic. This study proposes a hybrid anomaly detection pipeline that combines deep learning for representation learning with gradient-boosted decision trees for multiclass classification on structured traffic features. The pipeline applies Min–Max scaling to harmonize feature ranges and SMOTE to mitigate class imbalance, then uses a CNN–BiLSTM network to capture local patterns and temporal dependencies, followed by LightGBM as the final classifier. To enhance LightGBM performance, its hyperparameters are optimized using the Whale Optimization Algorithm (WOA). We evaluate the approach using accuracy, precision, recall, and F1-score in a multiclass setting. Experimental results show that the CNN–BiLSTM + LightGBM baseline achieves 0.92 accuracy with strong performance on most attack categories, while rare and ambiguous classes (notably Infiltration and Web Attack) remain challenging. With WOA-based optimization, overall accuracy increases to 0.93 and the macro-average F1-score improves from 0.88 to 0.89, with clearer gains for several time-dependent attack classes. While the proposed framework is designed with practical deployment in mind by leveraging a tree-based classifier on learned features, this study does not report system-level efficiency measurements (e.g., training time, inference latency, or memory footprint); these will be addressed in future work to substantiate computational feasibility in real server environments.

Read PDF

Similar papers

Open access Aug 2026

Unsupervised deep learning for IoT botnet detection in a surveillance VLAN via multi-source traffic, threat intelligence and honeypot corroboration

The increase in internet of thing devices, especially within VLANs in corporate networks, introduces significant security risks from advanced botnet attacks. Traditional signature-based detection methods struggle to identify encrypted, stealthy command-and-control traffic, while high false-positive rates overwhelm security teams with excess data. This study proposes an unsupervised deep-learning detection system using autoencoder (AE)-family models that learn normal traffic behaviors and identify anomalies through reconstruction error (MSE). An architecture comparison across five random seeds shows that a simple convolutional neural network (CNN)-AE performs similarly to a CNN-long short-term memory-AE (mean ROC-AUC difference of −0.0195 ± 0.0194); the hybrid is less stable and slower. The key contribution is the integrated, operational pipeline with a transparent evaluation approach, not architectural innovation. Tested on multi-source metadata—Wireshark, firewall logs, T-Pot honeypot—collected over fifteen days from a real enterprise VLAN, the system incorporates external threat intelligence (AbuseIPDB, OTX) and contextual behaviors through a multi-dimensional scoring system (MDSS), converting raw detections into prioritized risk scores. Under deployment, the model flags 3.75% of traffic as anomalous, with approximately 1% false positives on benign devices. Stress tests—including feature-leakage ablation, multiple seed runs, temporal holdouts, and clean-device testing—suggest a realistic record-level ROC-AUC of 0.72–0.78, compared to near-perfect (>0.99) results in saturated IP-partition testing, which is reported solely as an upper bound. The MDSS assigns risk tiers: 0 Very-High, 2 High, 26 Medium, and 37 Low for the 65 monitored devices. Anomalies are independently validated via two MITRE ATT&CK pipelines—one focused on honeypot/policy sources and one based solely on the model’s detections—both identifying the same techniques (T1071, T1573, T1046, T1090). This confirms that model decisions are not circularly based on the evaluation labels. With low-latency, GDPR/KVKK-compliant, metadata-only analysis and operational prioritization, this framework bridges the gap between deep-learning security solutions’ theoretical potential and their real-world enterprise application. It provides a transparent, reproducible methodology for proactive, scalable, and interpretable botnet detection.

Özkan Zeybek, H. Güler · 0 citations
Open access Jul 2026

A Hybrid Deep Learning Model for Efficient IOT Botnet Attack Detection

The proposed hybrid framework provides a robust, scalable, and reliable solution for real-time botnet attack detection, enhancing the security and resilience of modern IoT networks against evolving cyber threats.

Munagala Kusuma, M.ramesh · 0 citations
Open access Sep 2026

A Hybrid CNN–BiGRU Deep Learning Model for DDoS Attack Detection in Cybersecurity

Distributed Denial of Service (DDoS) attacks remain one of the most serious threats to modern network infrastructures, as they overwhelm systems with massive traffic and disrupt legitimate services. Traditional intrusion detection systems often struggle to detect complex and evolving attack patterns due to their reliance on manual feature engineering and limited learning capability. To address this issue, this study proposes a hybrid deep learning model that integrates Convolutional Neural Networks (CNN) and Bidirectional Gated Recurrent Units (BiGRU) for efficient DDoS attack detection. The CNN component is used to automatically extract important spatial features from network traffic data, while the BiGRU layer captures temporal dependencies and sequential patterns in the traffic flows. The proposed model is evaluated using the CICDDoS2019 dataset, which contains realistic benign and attack traffic across multiple DDoS categories. Experimental results demonstrate that the model achieves high performance in both binary and multiclass classification tasks, obtaining an accuracy of 99.82% for binary detection and 99.21% for multiclass classification. The results indicate that the hybrid CNN–BiGRU architecture effectively improves detection accuracy and provides a reliable approach for intelligent network intrusion detection systems.

Unknown authors · 0 citations
Review Open access 2024

Deep Learning-Based Network Traffic Classification for Next-Generation Networks

Network traffic classification has become essential for managing, securing, and optimizing next-generation networks (NGNs) that support IoT, cloud computing, SDN, NFV, and 5G technologies. Traditional classification methods based on port numbers and deep packet inspection are increasingly ineffective due to encrypted traffic, dynamic port allocation, and evolving application behaviors.This study presents a deep learning-based approach for intelligent network traffic classification. It reviews the evolution of traffic classification techniques and proposes a hybrid CNN-LSTM model that combines packet-level feature extraction with temporal sequence learning. The framework effectively captures spatial and temporal characteristics of network traffic, enabling accurate classification of both encrypted and non-encrypted flows. Experimental evaluation using benchmark datasets demonstrates that the proposed model outperforms conventional machine learning methods, achieving classification accuracy above 97%. Performance is validated using metrics such as accuracy, precision, recall, F1-score, and classification efficiency. The results highlight the potential of deep learning to support real-time traffic analysis, network security, QoS management, and resource optimization in modern communication networks. The proposed framework provides an intelligent and scalable solution for traffic classification in future autonomous and secure networking environments.

Chen Wei, Hiroshi Tanaka · 0 citations
Open access Aug 2026

Deep Learning-Based Network Intrusion Detection Using Hybrid CNN and LSTM Architecture

The findings indicate that hybrid deep learning techniques can improve network security by enhancing intrusion detection capability while reducing false alarms.

A. O. Jimoh-Mahmud, Abubakar Dayyabu, Abubakar Sadiq Idris et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.