Dual Knowledge Distillation for Heterogeneous Federated Continual Learning

Abstract

Federated Continual Learning (FCL) enables distributed clients to collaboratively learn a sequence of tasks while preserving data privacy and mitigating catastrophic forgetting. However, most existing FCL methods rely on the assumption that all clients share an identical model architecture, which is impractical in real-world federated systems due to diverse client resources. Although logit-based federated distillation provides a feasible solution for model-heterogeneous collaboration, the quality of aggregated logits would be severely degraded under continual learning scenarios, where model heterogeneity, data heterogeneity, and accumulated previous classes jointly introduce biased and unstable teacher signals. To address these challenges, we propose FedDKD, a dual knowledge distillation framework for model-heterogeneous federated continual learning. FedDKD jointly exploits logits and prototypes as complementary knowledge carriers to integrate sample-level decision knowledge with class-level semantic knowledge. Specifically, prototype ensemble distillation further provides semantic anchors for global model to align feature spaces and reduce the bias of logits aggregation on public data. Furthermore, FedDKD decouples the distillation of current-class and previous-class knowledge to balance plasticity and stability during continual learning, and introduces an inter-task prototype transfer mechanism to generate pseudo feature prototypes of previous classes without accessing raw previous-task data. Extensive experiments show that FedDKD achieves competitive performance compared to mainstream methods.