Scaling and Validating a Distributed Training and Retraining Pipeline for Network Intrusion Detection

Abstract

Network intrusion detectors must train at scale and be retrained as traffic drifts, which stresses pipeline reproducibility and distributed execution. We validate a batch training and retraining pipeline with distributed model fitting, a step towards end-to-end machine-learning operations (MLOps) rather than a complete MLOps implementation. It uses gradient-boosted decision trees with XGBoost-on-Spark on a bare-metal high-performance computing cluster; serving, a model registry, and drift monitoring are outside its scope. We evaluate it on two benchmarks, NSL-KDD and CIC-IDS2017, mapped to five traffic superclasses, along four axes: detection quality, class rebalancing, strong scaling from 1 to 16 nodes, and retraining time. Under a connection-grouped split, Macro-F1 reached 0.90 on CIC-IDS2017 (the more conservative of the evaluated splits), compared with 0.95 under a record-level split, while the ordering of rebalancing paradigms differed by dataset. On the ×512 replica (110 GB), model fitting achieved a 10.64-fold speedup at 16 nodes (95% CI 10.50–10.77), and parallel efficiency increased with data volume. Replacing 25 Gb/s Ethernet with 200 Gb/s InfiniBand over IPoIB left the model-fitting time statistically equivalent within a ±5% margin from 2 to 16 nodes (two one-sided tests over 30 repetitions per point). A complete retraining cycle took under half a per cent of an eight-hour maintenance window. We release the pipeline, curated results, and figure code as a reproducible package.