LAUA: Handling Missing Modalities and Unpaired Data in Multimodal Federated Learning

Abstract

Multimodal federated learning enables multiple clients to collaboratively train models from distributed multimodal data while preserving data privacy. In realistic federated settings, multimodal samples are often missing or unpaired, and cross-modal heterogeneity across clients can hinder stable optimization. Many existing federated multimodal methods attempt to mitigate modality missingness by generating synthetic paired data through data augmentation or generative models. However, they typically rely on paired supervision or treat client updates uniformly, making them brittle under modality missingness and client-level variability. To address these challenges, we propose a federated multimodal learning framework (LAUA) for learning from a mixture of unimodal and multimodal clients. On clients, LAUA aligns representations in a shared variational latent space, where KL regularization yields a principled and lightweight confidence signal for estimating uncertainty. Unimodal clients learn transferable representations via self-supervised objectives, while multimodal clients additionally leverage task supervision and incorporate an internal distillation component to enhance cross-modal consistency and stabilize local optimization. On the server, LAUA performs uncertainty-weighted aggregation that adaptively down-weights unreliable client updates. Experiments on various datasets show that LAUA substantially mitigates performance degradation under modality missingness across retrieval and regression tasks, attaining up to 20% relative improvement in MRR for retrieval and up to 24.3% relative improvement in MSE for regression.