Results indicate that directional mixture-to-constituent alignment provides an effective pre-training signal for polyphonic audio representation learning.
Abstract
Real-world audio typically contains concurrent sound events, yet standard self-supervised methods treat each clip as an isolated training unit without explicitly modeling mixture--constituent relationships. We propose Mixture-to-Constituent Latent Alignment (MiCLA), a two-stage self-supervised pre-training framework that uses synthetic log-mel mixing to align mixture representations with their constituent clips. Stage 1 structures single-clip representations via masked self-distillation and contrastive regularization. Stage 2 introduces bidirectional alignment: for a synthetic mixture, MiCLA suppresses the component aligned with one constituent and aligns the residual with the other, using projective residualization on patch tokens and sample-adaptive soft residualization on the global class token. In downstream evaluations, MiCLA achieves 43.2% mean average precision (mAP) on AS-20K and 50.3% mAP on AS-2M under fine-tuning, and achieves the highest mAP among evaluated models on both AudioSet settings under frozen Convex Gated Probing (CGP). Furthermore, a Joint Semantic Retrieval diagnostic shows that MiCLA improves simultaneous retrieval of both constituent classes over target-averaging baselines. These results indicate that directional mixture-to-constituent alignment provides an effective pre-training signal for polyphonic audio representation learning.
Text-to-audio (TTA) generation aims to synthesize realistic audio that faithfully reflects natural-language descriptions. Most TTA systems adopt a two-stage latent paradigm: an audio tokenizer is optimized for reconstruction and then frozen, after which a generative model is trained in the resulting latent space. Howev...
Run-Wu Shi, Kai Li, Yu-Jin Wang et al.· 0 citations
Continuous latent representations underlie recent diffusion-based lyrics-to-song and text-to-music systems, yet aligning lyric conditions with these latents remains difficult: existing systems resort to generator-side machinery such as sentence-level timestamps, auxiliary alignment objectives, or semi-autoregressive de...
Guo-Bin Ma, Yue-Peng Jiang, Da-Ke Guo et al.· 0 citations
CoVA is introduced, a lightweight framework that composes VAE features online into stage-wise guidance for flow-matching training, and uncovers a hidden confound in soft label conditioning: averaging class embeddings attenuates their magnitude, coupling semantic interpolation with conditioning strength.
Zhao-Wei Chen, Yi-Kai Ge, Lin-Ze Li et al.· 0 citations
Across ResNet12 and Conv64 in 5-way 1-shot and 5-shot evaluation, SAMPLESELECT gives the best OOD accuracy among the compared methods and improves the matched full-representation control by 4.90-8.38 percentage points.
Feng-Rui Liu, Ning Shen, Yi Li et al.· arXiv.org· 0 citations
Few-shot audio classification requires audio--language models to distinguish variable acoustic events from limited labeled data. PALM's main formulation adapts encoded class representations without changing pretrained encoders, but its class representations remain input-independent and classification uses audio feature...
Yu-Lin Sun, Sheng-Hua Li, Yong Dou et al.· 0 citations
Results demonstrate that slot-based representations capture transferable and compositional structure in audio scenes, enabling generalization across tasks and datasets.
Parthasaarathy Sudarsanam, Tuomas Virtanen· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.