Skip to content
Preprint

MiCLA: Mixture-to-Constituent Latent Alignment for Polyphonic Audio Representation Learning

Oct 2026 · 0 citations · 45 references
Computer Science Mathematics

TL;DR

Results indicate that directional mixture-to-constituent alignment provides an effective pre-training signal for polyphonic audio representation learning.

Abstract

Real-world audio typically contains concurrent sound events, yet standard self-supervised methods treat each clip as an isolated training unit without explicitly modeling mixture--constituent relationships. We propose Mixture-to-Constituent Latent Alignment (MiCLA), a two-stage self-supervised pre-training framework that uses synthetic log-mel mixing to align mixture representations with their constituent clips. Stage 1 structures single-clip representations via masked self-distillation and contrastive regularization. Stage 2 introduces bidirectional alignment: for a synthetic mixture, MiCLA suppresses the component aligned with one constituent and aligns the residual with the other, using projective residualization on patch tokens and sample-adaptive soft residualization on the global class token. In downstream evaluations, MiCLA achieves 43.2% mean average precision (mAP) on AS-20K and 50.3% mAP on AS-2M under fine-tuning, and achieves the highest mAP among evaluated models on both AudioSet settings under frozen Convex Gated Probing (CGP). Furthermore, a Joint Semantic Retrieval diagnostic shows that MiCLA improves simultaneous retrieval of both constituent classes over target-averaging baselines. These results indicate that directional mixture-to-constituent alignment provides an effective pre-training signal for polyphonic audio representation learning.

View source

Similar papers

Preprint Sep 2026

UNITE-AUDIO: Joint Learning of Continuous Tokenization and Latent Flow Matching for Text-to-Audio Generation

Text-to-audio (TTA) generation aims to synthesize realistic audio that faithfully reflects natural-language descriptions. Most TTA systems adopt a two-stage latent paradigm: an audio tokenizer is optimized for reconstruction and then frozen, after which a generative model is trained in the resulting latent space. Howev...

Run-Wu Shi, Kai Li, Yu-Jin Wang et al. · 0 citations
Preprint Oct 2026

MSVAE: Rethinking Musical Semantics in Continuous Latent Spaces

Continuous latent representations underlie recent diffusion-based lyrics-to-song and text-to-music systems, yet aligning lyric conditions with these latents remains difficult: existing systems resort to generator-side machinery such as sentence-level timestamps, auxiliary alignment objectives, or semi-autoregressive de...

Guo-Bin Ma, Yue-Peng Jiang, Da-Ke Guo et al. · 0 citations
Preprint Oct 2026

CoVA: Composing VAE Representations for Generative Learning

CoVA is introduced, a lightweight framework that composes VAE features online into stage-wise guidance for flow-matching training, and uncovers a hidden confound in soft label conditioning: averaging class embeddings attenuates their magnitude, coupling semantic interpolation with conditioning strength.

Zhao-Wei Chen, Yi-Kai Ge, Lin-Ze Li et al. · 0 citations
Preprint Oct 2026

DF-PALM: Dynamic Fusion Prompt Learning for Audio--Language Models

Few-shot audio classification requires audio--language models to distinguish variable acoustic events from limited labeled data. PALM's main formulation adapts encoded class representations without changing pretrained encoders, but its class representations remain input-independent and classification uses audio feature...

Yu-Lin Sun, Sheng-Hua Li, Yong Dou et al. · 0 citations
Preprint Oct 2026

Self-Supervised Object-Centric Representation for Multi-Task Audio Analysis

Results demonstrate that slot-based representations capture transferable and compositional structure in audio scenes, enabling generalization across tasks and datasets.

Parthasaarathy Sudarsanam, Tuomas Virtanen · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.