FedPivot: Bridging Heterogeneous Multimodal Pre-training via Federated Post-Alignment
Abstract
The rapid evolution of Multimodal Large Language Models (MLLMs) is bottlenecked by the saturation of high-quality public data, while vast amounts of heterogeneous multimodal data remain inaccessible in privacy-sensitive silos. Federated Learning (FL) offers a promising solution to leverage these distributed resources while preserving privacy, but existing research focuses predominantly on fine-tuning, leaving the foundational pre-training phase largely unexplored. We argue that the core of MLLM pre-training lies in establishing effective cross-modal alignment, which bridges visual and textual representations into a unified semantic space. Motivated by this insight, we formally introduce the Federated Post-Alignment (Fed-PA) task, which studies cross-modal alignment under federated MLLM pre-training by collaboratively training a shared cross-modal projector. We identify two critical challenges in this setting: (i) parameter interference in aggregating local projectors; and (ii) gradient oscillations in one-pass collaborative SGD. To address these challenges, we propose FedPivot, a pioneering framework for federated post-alignment. FedPivot employs Canonical Reliability-Aware Aggregation (CRA), which constructs a canonical space to decompose client projectors into a shared alignment basis and client-specific coefficients, then performs reliability-weighted fusion to suppress parameter interference. Furthermore, FedPivot introduces Orthogonality-Preserved Momentum (OPM), which applies momentum to the shared alignment basis via orthogonal projection, accumulating historical optimization directions while preserving geometric structure. We construct four federated multimodal pre-training scenarios based on public datasets, and extensive experiments validate that FedPivot significantly outperforms existing baselines.