A Data-Centric Network Traffic Dataset for Anomaly Detection: Construction, Reproducible Pipeline, and Technical Validation

A survey of intrusion detection datasets for communication networks

This paper presents a comprehensive survey of datasets used for intrusion detection in communication networks, examining 124 datasets and characterizing each across 20 key dimensions, such as attack diversity, temporal coverage, data formats, and availability. We systematically analyze how these datasets support research on intrusion detection systems (IDS) in networking environments by reviewing detection approaches, machine and deep learning models, feature selection techniques, traffic analysis tools, and performance metrics. Beyond descriptive comparison, the survey analyzes datasets according to their application domains, readiness for modern AI-driven IDS architectures, realism, quality and benchmarking risks, and sustainability and reproducibility. Our in-depth comparative analysis reveals critical gaps—such as limited real-world traffic representation, inconsistent documentation, dataset bias, evaluation risks, and underrepresentation of emerging attack vectors—and provides actionable recommendations for dataset standardization, high-fidelity data collection, improved labeling, quantitative dataset assessment, and privacy-preserving sharing. Furthermore, we introduce SHIELD, a continuously evolving online repository that supports centralized dataset discovery, comparison, and selection. This survey aims to guide researchers in selecting appropriate datasets for evaluating IDS in communication networks and to inform future efforts toward more realistic, scalable, reliable, and reproducible intrusion detection research.

A domain-agnostic explainable framework for network attack detection across diverse traffic datasets

An explainable deep learning framework evaluated across multiple heterogeneous cyber attack datasets, including Kitsune, Server-Based network data,enterprise logs, and Malware Traffic datasets, demonstrating the effectiveness of the proposed framework in handling heterogeneous network traffic while providing interpretable insights into model predictions.

Autoencoding-Based Self-Supervised Learning for Enhanced Representation of Network Traffic Patterns

Network Traffic Monitoring and Analysis (NTMA) is increasingly important given the growing volume of network data and the associated cyber threats. Effective NTMA involves analyzing data packets for performance optimization, security, and policy compliance. In recent years, Machine Learning (ML) has shown high performance in this domain; however, traditional ML methods rely heavily on labeled data, which is costly and scarce. This paper proposes AE-SSL (Autoencoding-Based Self-Supervised Learning), a denoising autoencoder framework that enhances the representation of IoT-centric network traffic patterns using unlabeled data. AE-SSL employs two complementary pretext tasks tailored to tabular data (binary mask prediction and corrupted-feature reconstruction), and improves the performance of various classification models, including traditional and deep learning-based models. In our IoT-centric evaluation on two datasets, AE-SSL significantly improves classification metrics compared to supervised learning approaches. For example, on the ACI IoT 2023 dataset (100 pretraining epochs), AE-SSL improves AdaBoost macro-precision from 70.3% to 83.0% and SVC accuracy from 86.0% to 96.4%. On the CICIoT 2023 dataset (100 epochs), AdaBoost accuracy improves from 95.1% to 96.9%, and Logistic Regression accuracy from 73.7% to 76.2%. Additionally, an ablation study on key parameters reveals their impact on model performance, providing insights into optimizing self-supervised learning for network traffic analysis. These findings point to the potential of self-supervised techniques for IoT-centric NTMA; generalization to enterprise, mobile, and cloud traffic remains to be established and is left to future work.

Adaptive Methods and Tools for Shallow Packet Inspection in Anomaly Detection within Encrypted Network Traffic

Core Traffic Feature Identification and Cross-Model Validation for Lightweight DDoS Detection

DDoS attacks exhaust target resources through large numbers of concurrent connections, requests, or packets, and remain a major threat to service availability. Machine-learning studies on public benchmarks such as CSE-CIC-IDS2018 often report high classification performance, but high scores alone do not explain which traffic statistics are actually used by the models. They also do not indicate how many features should be collected in lightweight deployment to balance detection performance and computational cost. This paper uses DDoS-related traffic from CSE-CIC-IDS2018 and constructs Benign/DDoS binary detection and Benign/LOIC-HTTP/HOIC multiclass recognition tasks. RandomForest is first used to rank the features, and RandomForest, XGBoost, and LightGBM are then used for cross-model validation. The experiments combine top-k analysis, a Random-5 low-dimensional baseline, remove-top-5 ablation, and multi-seed stability analysis. The results show that top-5 is a near-saturated point for binary detection, while top-2 already forms a compact high-performance solution for multiclass recognition and fixed top-5 further adds packet-length information. After reducing 66 statistical features to 5, the absolute F1 changes of the three models in the binary task are all below 0.0003, and the multiclass Macro-F1 remains close to the full-feature setting. Overall, the selected features should not be interpreted as isolated variables; instead, they form a compact forward-traffic feature cluster. This cluster covers initial window behavior, forward packet volume, forward byte volume, and packet-length characteristics, providing discriminative information for lightweight DDoS detection and recognition of major attack subtypes.

Network log anomaly detection and evaluation based on unsupervised deep learning

With the rapid development of Internet technology and the deepening of digital transformation, network log data has experienced explosive growth, and network attacks are becoming increasingly complex and covert. Traditional manual analysis and detection methods, which rely on labeled data and ignore temporal and semantic features, are no longer sufficient to meet the needs of modern network security. This study aims to construct an efficient unsupervised deep learning framework for network log anomaly detection, addressing the problems of scarce anomaly samples, difficulty in labeling, and insufficient detection performance of traditional methods. First, network log data undergoes preprocessing such as cleaning, integration, and feature extraction, employing hybrid sampling to balance data distribution. Then, a model combining an autoencoder and generative adversarial network (GAN) is constructed, and hyperparameters are adjusted through intelligent optimization algorithms. Anomalies are identified using a dual mechanism of reconstruction error and discriminant loss. Finally, the model performance is validated on three public datasets—HDFS, BGL, and Thunderbird— based on multiple metrics including accuracy, F1 score, and AUC. Experimental results show that the proposed framework's autoencoder model achieves an F1 score of 0.92 and an AUC of 0.95, significantly outperforming the GAN model and traditional methods. It effectively balances detection accuracy and recall while reducing false positives. This research does not rely on a large amount of labeled data and can automatically identify potential cybersecurity threats, providing intelligent and automated monitoring methods for cybersecurity operations and promoting the practical development of unsupervised deep learning in the field of anomaly detection.