Skip to content
Book Open access

FedPivot: Bridging Heterogeneous Multimodal Pre-training via Federated Post-Alignment

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · 0 citations · 48 references

Abstract

The rapid evolution of Multimodal Large Language Models (MLLMs) is bottlenecked by the saturation of high-quality public data, while vast amounts of heterogeneous multimodal data remain inaccessible in privacy-sensitive silos. Federated Learning (FL) offers a promising solution to leverage these distributed resources while preserving privacy, but existing research focuses predominantly on fine-tuning, leaving the foundational pre-training phase largely unexplored. We argue that the core of MLLM pre-training lies in establishing effective cross-modal alignment, which bridges visual and textual representations into a unified semantic space. Motivated by this insight, we formally introduce the Federated Post-Alignment (Fed-PA) task, which studies cross-modal alignment under federated MLLM pre-training by collaboratively training a shared cross-modal projector. We identify two critical challenges in this setting: (i) parameter interference in aggregating local projectors; and (ii) gradient oscillations in one-pass collaborative SGD. To address these challenges, we propose FedPivot, a pioneering framework for federated post-alignment. FedPivot employs Canonical Reliability-Aware Aggregation (CRA), which constructs a canonical space to decompose client projectors into a shared alignment basis and client-specific coefficients, then performs reliability-weighted fusion to suppress parameter interference. Furthermore, FedPivot introduces Orthogonality-Preserved Momentum (OPM), which applies momentum to the shared alignment basis via orthogonal projection, accumulating historical optimization directions while preserving geometric structure. We construct four federated multimodal pre-training scenarios based on public datasets, and extensive experiments validate that FedPivot significantly outperforms existing baselines.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.