Personalized open-set intrusion detection with dynamic class discovery for heterogeneous Internet of Things

Abstract

In Internet of Things(IoT) environments, most intrusion detection systems are trained under a static closed-set assumption, where attack categories are predefined, and the model is deployed without mechanisms for continuous adaptation. However, severe device heterogeneity and rapidly evolving attacks undermine this assumption, making a single global model unreliable across diverse client distributions and limiting the ability of existing methods to further analyze and incorporate unknown attacks. To address these limitations, we propose a personalized open-set intrusion detection framework for IoT environments. We propose a class-indexed deep k-nearest neighbor open-set detector that performs class-conditional retrieval in the feature space to identify unknown attacks while reducing false rejections of known classes. We further propose a novel-class discovery algorithm that clusters samples rejected as unknown to uncover emerging attack patterns, which are then gradually integrated into the known-class set via incremental training to expand the known category set. To cope with heterogeneous non-independent and non-identically distributed(non-IID) client data and privacy constraints, we adopt a personalized federated learning scheme that shares global knowledge while preserving client-specific decision behavior. Finally, we design a client-oriented evaluation protocol that separately assesses known-class recognition and unknown rejection on each client’s local distribution. Experiments across multiple scenarios demonstrate improved detection of unknowns, effective absorption of new classes, and stable personalized performance.Methodologically, the proposed class-indexed open-set detection, novel-class discovery, and incremental learning mechanisms support the detection and integration of unseen attacks. Practically, they are incorporated into a personalized federated framework for adaptive intrusion detection in heterogeneous non-IID IoT environments.