Beyond Detection Accuracy: Measuring Explanation Cost, Stability, and Utility for Resource-Aware IoT Intrusion Detection

A Reliability-Aware Edge–Cloud Framework for Early Intrusion Detection in IoT Networks

Gateway-resident intrusion detection can act before IoT traffic reaches cloud services, but early decisions are based on incomplete flow prefixes. This paper presents a reliability-aware edge–cloud framework that treats early detection as a sequential routing problem. At each checkpoint, a lightweight gated recurrent unit (GRU) maps causal packet-prefix features to a malicious-probability estimate. Temperature scaling, asymmetric benign and malicious thresholds, and an eight-packet minimum-evidence gate determine whether a flow exits locally, remains under observation, or is sent for cloud refinement. Short and unresolved flows are classified by regularized logistic regression using a compact 97-feature causal representation. The edge model contains 19,777 parameters, and each cloud submission carries 388 bytes of float32 features. The principal evaluation uses all 309 CIC-IoT-2023 PCAP files under four outer PCAP-disjoint folds, with separate edge-training, calibration, cloud-development, and final-test roles. Across 2,286,754 pooled out-of-fold flows with 88.54% malicious prevalence, the framework resolves 422,190 flows at the edge and routes 1,864,564 for cloud refinement, reducing cloud submissions by 18.46%. The final policy attains 4.47% FPR, 1.89% FNR, 96.82% balanced accuracy, and 98.76% F1 score. Observation-budget analysis identifies 32 packets as a corpus-specific compromise, whereas controlled delays in post-eight-packet information expose the limits of short-prefix detection. On the balanced CICIDS2017 test set, in-domain development attains 97.03% balanced accuracy; zero-shot transfer falls to 86.30%, and target-calibration-only adaptation improves it to 91.65%. Ablation results identify the minimum-evidence gate and cloud-refinement stage as the main reliability controls. Benign false alarms, delayed post-eight-packet information, cross-dataset shift, and scenario/file-level labels remain the principal limitations.

Reliability auditing of explanations for machine-learning-based intrusion detection systems

Machine-learning-based intrusion detection systems can learn nonlinear and interaction-based traffic patterns that are difficult to capture using static rules, but their predictions remain difficult to interpret reliably in analyst-facing cybersecurity workflows. This paper proposes a unified quantitative framework for auditing the reliability of Explainable Artificial Intelligence (XAI) in intrusion detection systems. The framework combines SHAP attributions with permutation-based functional importance, SHAP-permutation rank agreement, sufficiency and comprehensiveness retraining tests, retraining-stability analysis, spurious-feature injection, and a composite XAI reliability radar profile. The framework is evaluated on UNSW-NB15 and IoT-ToN using XGBoost, Logistic Regression, and Random Forest. Final performance is reported on untouched test dataset after training-only cross-validation and model selection. On UNSW-NB15, Random Forest and XGBoost achieved comparable held-out performance, with ROC AUC, PR AUC, and F1-scores of 0.9862, 0.9934, and 0.9231 for Random Forest, and 0.9858, 0.9932, and 0.9217 for XGBoost. Logistic Regression performed lower, with scores of 0.9695, 0.9774, and 0.9037. On IoT-ToN, XGBoost and Random Forest achieved near-ceiling performance, with ROC AUC values of 0.9994 and F1-scores of 0.9867 and 0.9879, respectively, while Logistic Regression degraded substantially, with ROC AUC of 0.8502 and F1-score of 0.6081. The reliability results show that high predictive performance does not automatically imply trustworthy explanations. Logistic Regression produced the most stable explanations, but its weaker detection performance limited its practical suitability. Among the high-performing models, XGBoost provided the strongest balance on IoT-ToN, while Random Forest provided the strongest balance on UNSW-NB15. These findings demonstrate the need to evaluate XAI-enabled intrusion detection using both predictive metrics and quantitative explanation-reliability diagnostics.

Feature Selection Versus Feature Extraction for IoT Intrusion Detection Under Temporal Evaluation

LARO-IDS: Family-leakage-aware robust multi-objective optimization for model selection in IoT intrusion detection

Machine learning-based intrusion detection systems (IDS) are commonly selected based on conventional validation or development metrics, although such criteria may not sufficiently reflect robustness against unseen attack families or suitability for resource-constrained Internet of Things and edge environments. This study proposes learned acquisition and reconstruction optimization (LARO)—IDS (LARO-IDS), a family-leakage-aware robust multi-objective optimization framework for model selection in Internet of Things intrusion detection. Instead of selecting the model that only maximizes conventional predictive performance, LARO-IDS jointly considers development macro-F1, mean cross-family robustness, worst-family behavior, robustness variability, and prediction latency in the candidate-selection objective, while training time and model size are retained as additional deployment-cost indicators for final comparison. Candidate models were evaluated using a model-selection evaluation subset and a leave-one-attack-family-out robustness protocol, then ranked using a weighted-sum scalarization of normalized objectives, with the results further supported by Pareto-efficiency analysis. Experiments on the CICIoT2023 dataset show that conventional score-based selection favors RF_03_regularized, which achieved the highest macro-F1. In contrast, LARO-IDS selects RF_01_fast, which preserves nearly identical predictive performance, with only a −0.0015 macro-F1 difference, while achieving slightly higher mean cross-family F1 scores across attack families. The LARO-selected model also reduces training time by 49.49%, prediction latency by 46.36%, and model size by 50.12% compared with the conventionally selected model. Sensitive analysis of objective weights further shows that RF_01_fast remains selected under balanced, performance-priority, robustness-priority, and edge-priority scenarios. These results demonstrate that robust IDS model selection should integrate family-leakage-aware robustness and latency-aware deployment cost rather than relying solely on conventional predictive performance.

Benchmark-Shift-Aware Intrusion Detection for Evolving Network Traffic: Cross-Dataset Generalization, Calibrated Alerting, and Score-Orientation Diagnostics

Modern intrusion detection systems (IDSs) are often evaluated under matched training and test conditions, whereas deployment environments involve changing traffic distributions, heterogeneous feature-generation pipelines, and shifting attack prevalence. This study investigates benchmark-shift-aware intrusion detection through harmonized cross-dataset evaluation of HIKARI-2021, CICIDS2017, and a CICIoT2023 sample subset. Two payload-free feature spaces are constructed: Rich-64 for detailed HIKARI-2021/CICIDS2017 analysis and Minimal-13 for three-way comparison. Using XGBoost, a supervised Transformer, and a masked-feature self-supervised Transformer, we evaluate discrimination, calibration, threshold transfer, alert-budget behavior, chronological robustness, and score-orientation stability. Across five in-domain XGBoost settings, observed false-positive rates were 4.94–5.40%, and F1-scores ranged from 0.507 to 0.995. Under strict Rich-64 HIKARI-2021-to-CICIDS2017 transfer, all models had zero recall at source-derived thresholds, with two showing inverted score orientation. In the reverse direction, XGBoost reached an 18.9% target false-positive rate, while a nominal 5% target-side alert budget yielded F1 = 0.112. Chronological evaluation further showed that improved ranking metrics did not guarantee stable validation-derived operating behavior. The study provides a reproducible diagnostic framework for evaluating IDS robustness under evolving benchmark conditions.

Explainable Machine Learning for DDoS Attack Detection with Physical Network Validation

Distributed Denial-of-Service (DDoS) attacks remain one of the most disruptive threats to network infrastructure, yet many machine learning (ML)-based detection studies report only offline benchmark performance without verifying whether that performance holds under real network conditions. This study evaluates two explainable ML classifiers, XGBoost and Random Forest, for DDoS detection and examines whether their near-perfect offline accuracy translates into reliable physical-network operation. The study combines offline benchmarking on the CIC-DDoS2019 dataset (293,485 flows) with physical-network validation using a working Intrusion Detection System (IDS) prototype under a controlled SYN-flood attack. Session-disjoint stratified sampling prevented flow-level leakage across attack sessions, while SHapley Additive exPlanations (SHAP) interpreted global and local feature importance. Offline, both classifiers achieved near-perfect performance (accuracy 99.99% for XGBoost, 99.98% for Random Forest; F1 = 0.9999; ROC-AUC up to 1.0000), with no statistically significant difference between them (McNemar's exact test, p = 0.2188), though XGBoost achieved approximately 3.69 times higher inference throughput (1,819,816 flows/s). SHAP identified Min Packet Length, Fwd Packet Length Min, Inbound, Protocol, and Init_Win_bytes_forward as the most influential features. In physical deployment, however, the IDS prototype flagged 6.26% of captured flows (6,935 of 110,762) as ATTACK during the SYN-flood test, and a separate 397-flow ambient-benign subset yielded a 6.80% false positive rate (95% Wilson CI: 4.72–9.71%), with short-duration SSDP/UPnP-style UDP control traffic accounting for 70% of observed false positives. This gap shows that near-perfect offline accuracy does not guarantee low false positives in real deployment, indicating that offline benchmarks alone are insufficient for validating IDS readiness.