Skip to content

Binding Multiple Modalities via Multimodal Wasserstein Barycenter

Sep 2026 · 2 citations · 72 references
Computer Science

TL;DR

BaryBind is presented, which aims to transport the specific modality towards the Wasserstein barycenter (WB) optimized across all modalities and introduces a volumetric alignment objective to establish a unified semantic space around the WB embedding.

Abstract

Multimodal learning beyond two modalities commonly leverages a specific modality (e.g., text) to bind other modalities. However, how to establish a more balanced representation space that approximates shared semantics while respecting the holistic geometry of $n$-modal data remains challenging. In this work, we present BaryBind, which aims to transport the specific modality towards the Wasserstein barycenter (WB) optimized across all modalities and introduces a volumetric alignment objective to establish a unified semantic space around the WB embedding. Specifically, we project specific modalities to the WB, which minimizes the average Wasserstein distances to multimodal distributions and serves as the anchor for subsequent alignment. We then construct a barycenter simplex, whose volume is taken as a similarity metric for global alignment centered at the WB. Experiments show that BaryBind achieves competitive performance in text-video-audio retrieval, classification, videoQA, and cross-modal generation tasks, along with robustness under modality absence and scalability to more than three modalities. Code is released at https://github.com/xl-tang3/BaryBind.

View source

Similar papers

Sep 2026

Learning curvature-aware multimodal representations with bicomplex embeddings

CurvBi (Curvature-aware Bicomplex), a novel framework for multimodal knowledge graph completion that unifies modality alignment and relational modeling under a coherent mathematical foundation, is proposed.

Thanh Le, Vy Cao, Duong-Khang Au et al. · 0 citations
Preprint Aug 2026

CrossFeat: Bridging Imaging Modalities in Feature Descriptor Space

This work proposes CrossFeat, a framework that enables an existing monomodal descriptor to operate across modalities, and introduces a geometry-appearance disentanglement such that only appearance is altered while the geometric properties are preserved.

Paul-Werner Schneider, Nazim Haouchine · 1 citation
Sep 2026

Beyond Spatial-Domain Supervision: A Relation Constrained Space for Multi-Modal Image Fusion

Multi-modal image fusion (MMIF) aims to form a single image by integrating shared information, preserving complementary cues, and coordinating cross-modal conflicts across modalities. However, due to the absence of ground-truth fused images, existing MMIF supervision commonly uses spatial-domain sources or gradient var...

Ze-Yu Wang, Ming-Yu Ge, Hai-Yu Song et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Do More Modalities Always Help? A Geometric Perspective on Missing-Modality Robustness

Geodesic Unlearning (GU), a lightweight parameter-editing method that leverages Grassmannian subspace geometry for structured subspace correction to improve missing-modality robustness, is proposed and suggested as a practical route for robustness correction.

Song-Yuan Sui, Zhen Tan, Mohan Zhang et al. · 0 citations
#computer vision Preprint Oct 2026

UNIVERS: Learning UNIversal Voxel-wise Embedding via Randomized Synthesis for generalizable one-shot landmark localization

One-shot landmark localization in volume scans with varying fields of view (FOVs) is challenging. While existing self-supervised learning methods have shown promising performance, they rely on contrastive learning built upon augmented views of the same image, which inherently limits their generalization in cross-modali...

De-Shuai Yu, Qing-Song Zhao, Yan-Bing Yue et al. · 0 citations
#machine learning Preprint Oct 2026

TartanMatch: Towards Universal Dense Correspondence Across Modalities

Robotic platforms combine multi-modal sensors to operate across diverse conditions, but most dense correspondence methods are limited to RGB-RGB or a small set of modality pairs. Training a model for each modality pair scales poorly with the number of sensors and forgoes potential positive transfer across modality pair...

Hyeokjoon Kwon, Ji-Ting Cai, Ruo-Gu Li et al. · 0 citations

Related blog posts

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.