Machine Learning-Based Anomaly Detection in Post-Quantum TLS 1.3 Traffic: A Comparative Analysis on the CIC-PQC_OAV v1 Dataset

Abstract

The integration of post-quantum cryptographic algorithms into the TLS 1.3 protocol may alter the behavioral characteristics of network traffic and affect the performance of existing anomaly detection systems. While secure communication in earlier uses of TLS largely relied on classical key exchange mechanisms, the inclusion of post-quantum algorithms in TLS 1.3 sessions may lead to different traffic behaviors. Considering this change, this study performs machine learning-based anomaly detection on the CIC-PQC_OAV v1 dataset. Decision Tree, K-Nearest Neighbor, Support Vector Machine, Random Forest, and XGBoost algorithms were applied using the same training, validation, and test partitions. The models were compared in terms of Accuracy, Precision, Recall, F1-Score, and ROC-AUC. The experimental results show that Decision Tree and Random Forest achieved the most balanced performance, with an Accuracy of 97.79% and an F1-Score of 90.31%. XGBoost stood out in terms of class separability, achieving a ROC-AUC value of 98.35%. The findings indicate that post-quantum TLS 1.3 traffic presents a different and more challenging classification structure than traditional network traffic. Therefore, anomaly detection systems should be re-evaluated by considering the traffic characteristics specific to post-quantum network environments.