Friend-Safe Adversarial Attack for selective evasion in personalized federated learning

Abstract

Federated Learning (FL) enables collaborative model training across distributed clients while preserving data privacy; however, the personalization of client models in non-independent and identically distributed (non-IID) settings creates an unexplored attack surface where adversarial examples can selectively fool specific target models while preserving correct predictions on designated friend models. In this paper, we introduce the Friend-Safe Adversarial Attack, a novel attack framework designed for personalized federated learning environments that formulates the generation of adversarial examples as a constrained optimization problem balancing two competing objectives: maximizing attack success on target (enemy) models while minimizing impact on friend models. We propose a momentum-based iterative optimization algorithm that combines Carlini-Wagner style margin loss for attacking targets with Kullback-Leibler divergence and cross-entropy losses for preserving friend model predictions. Extensive experiments on MNIST, CIFAR-10, and SVHN datasets demonstrate that, under the primary local fine-tuning personalization configuration with exact friend-model access, our Friend-Safe Attack achieves at least 88% attack success rate on enemy models while maintaining at least 81% accuracy on friend models, significantly outperforming standard Projected Gradient Descent (PGD) attacks, which reduce friend accuracy to near 0% on CIFAR-10 and SVHN and to 52.2% on MNIST. Additional experiments on FedPer, multi-friend coalitions, surrogate friend access, and Tiny-ImageNet and CIFAR-100, together with a sweep over all target–friend pairings, confirm that the selective-evasion property generalizes across personalization strategies, friend coalition sizes, attacker information levels, and recognition task complexity; the surrogate-access evaluation in particular indicates that the headline friend-preservation numbers reduce substantially when the attacker lacks exact friend-model access, and we therefore frame those numbers as the primary-scenario upper bound throughout the paper. Because local fine-tuning never returns the personalized models to the server, we additionally evaluate an attacker restricted to the friend-model information that a passive aggregation server observes, namely the friend’s final-round upload, the aggregated global model, and an extrapolation of the two; this access level preserves 92 to 96 percent of the friend’s predictions on MNIST, 70 to 74 percent on SVHN, and 37 to 43 percent on CIFAR-10, placing it between exact access and surrogate access. Operating points in this passive-server evaluation are selected on a held-out validation subset using only the model the attacker holds; applying the same protocol to surrogate access is what separates the two secondary levels. Our findings reveal critical security implications for federated learning systems and highlight the necessity for defense mechanisms that account for selective adversarial threats in multi-model environments.