MAPFL: Modular Adaptive Personalised Federated Learning From Visual Sentiment to Physical Activity

Abstract

Human‐centric sensing tasks such as visual sentiment analysis and smartphone‐based human activity recognition (HAR) exhibit strong client heterogeneity from feature, label, and concept shifts. Among these, concept shift—how individual users interpret and generate human‐centric signals—is particularly challenging yet underexplored in federated learning. We propose MAPFL (Modular Adaptive Personalised Federated Learning), a personalised federated framework that performs encoder‐level personalisation while preserving standard communication. Each client model is decomposed into a shared encoder and a local classifier head, and equipped with both global and personalised encoders. A gradient‐aligned personalisation controller learns client‐specific mixing coefficients that adaptively balance the two encoders based on the alignment of their gradients, allowing each client to decide how much to share versus personalise at the representation level. This design supports both image‐ and sensor‐based tasks. On an affective‐computing benchmark built from three in‐the‐wild visual sentiment datasets (ARTphoto, ABSTRACT, PARA) and on a smartphone‐based HAR benchmark derived from UCI HAR, MAPFL improves AUC and accuracy by up to 3.3% over strong federated and personalised baselines and converges faster and more stably. These results show that gradient‐aligned encoder personalisation is an effective and practical mechanism for handling multi‐dimensional heterogeneity in human‐centric sensing.