This work introduces a training-free approach that leverages Non-negative Matrix Factorization (NMF) to co-factorize audio and visual features from pre-trained models so as to reveal shared interpretable concepts and establishes state-of-the-art performance in unsupervised sound-prompted segmentation.
Abstract
Large-scale pre-trained audio and image models demonstrate an unprecedented degree of generalization, making them suitable for a wide range of applications. Here, we tackle the specific task of sound-prompted segmentation, aiming to segment image regions corresponding to objects heard in an audio signal. Most existing approaches tackle this problem by fine-tuning pre-trained models or by training additional modules specifically for the task. We adopt a different strategy: we introduce a training-free approach that leverages Non-negative Matrix Factorization (NMF) to co-factorize audio and visual features from pre-trained models so as to reveal shared interpretable concepts. These concepts are passed on to an open-vocabulary segmentation model for precise segmentation maps. By using frozen pre-trained models, our method achieves high generalization and establishes state-of-the-art performance in unsupervised sound-prompted segmentation, significantly surpassing previous unsupervised methods.
This work proposes a Training-Inference Co-optimization (TIC) framework that systematically enhances both stages of the retrieval pipeline, and achieves state-of-the-art performance among embedding-based techniques, significantly outperforming strong baseline models-even when utilizing a more lightweight text encoder.
CrossEdit is presented, a unified omni-modal editing model for images, audio, and video that performs AV movie scene edits zero-shot through cross-modal transfer and proposes AV-FES, a metric that jointly scores instruction following and consistency.
William Chen, Prem Seetharaman, Ke Chen et al.· 0 citations
A self-supervised model that integrates a time-frequency decoupled (TF-D) stem with masked latent reconstruction with the strongest overall transfer results among the internal variants and a competitive balance among representation quality, encoder scale, and inference efficiency is proposed.
Jie Xu, Yu-Hao Dai, Zhi-Feng Wang· Signal, Image and Video Proc...· 0 citations
In this paper, we study the task of automatic music upmixing, wherein a system predicts spatial mixing parameters from a multi-stem recording. Different from existing methods that rely on task-specific music encoders, we approach this task via audio language model (ALM) post-training, leveraging rich representations fr...
Zi-Xun Guo, Calvin Murdock, Sanjeel Parekh et al.· 0 citations
Visual object removal can eliminate a target from video frames, yet its acoustic trace persists in the soundtrack, causing obvious audio-visual inconsistency. Existing video inpainting models operate solely on pixels, while audio editing models, especially for the sound removal task, are typically driven by text and th...
Xin-Yue Guo, Jian-Xuan Yang, Dai-Guo Zhou et al.· 0 citations
Results indicate that directional mixture-to-constituent alignment provides an effective pre-training signal for polyphonic audio representation learning.
Kohei Yamamoto, Kosuke Okusa· 0 citations
Related blog posts
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026