Results demonstrate that slot-based representations capture transferable and compositional structure in audio scenes, enabling generalization across tasks and datasets.
Abstract
This paper studies the capability of an object-centric audio representation to support multiple downstream tasks within a unified framework. The object-centric model is trained in a self-supervised setup using slot attention by reconstructing Audio Spectrogram Transformer (AST) patch embeddings from a set of latent slots. It is trained on synthetic audio mixtures, which provides controlled multi-source audio scenes. These slot representations are evaluated for the classification, detection, and separation of sound events by training task specific output layers. On the classification task, the model achieves an F1-score of 0.8, comparable to clip-level AST embeddings which achieve 0.7. For sound event detection, it reaches a macro F1-score of 0.7, outperforming clip-level AST embeddings at 0.4 and matching AST patch embeddings at 0.7, which encode finer time-frequency information. In source separation, it achieves an SI-SDR improvement (SI-SDRi) of 18.4 dB. Furthermore, in a zero-shot source separation setting, the model achieves an SI-SDRi of 14.5 dB. These results demonstrate that slot-based representations capture transferable and compositional structure in audio scenes, enabling generalization across tasks and datasets.
Audio classification across speech-command and environmental-sound tasks requires both fine temporal cues and structured spectral context, yet many high-performing systems rely on external pretraining. We propose CATF-Net, a supervised framework trained from random initialization using only the labeled training data of...
Zhi-Cheng Pei, Jiang-Peng Li, Tong An et al.· IEEE Access· 0 citations
Prior audio-visual self-supervised learning methods rely on mechanisms such as EMA target encoders, prediction heads, reconstruction decoders, and contrastive losses. We introduce LeAVJEPA, the first audio-visual encoder trained under LeJEPA's collapse-free objective. A single early-fusion Vision Transformer processes...
Benjamin Robson, Santeri Mentu, Wen-Shuai Zhao et al.· 0 citations
This paper presents a novel Cross-Modal Distillation Contrastive Language-Audio Pretraining (CDCLAP) framework to address the challenges of zero-shot audio event classification (ZAEC) under limited hardware resources. The proposed method transfers visual knowledge from an ImageNet-pretrained MobileNetV3 to a lightweigh...
Results indicate that directional mixture-to-constituent alignment provides an effective pre-training signal for polyphonic audio representation learning.
Speaker verification (SV) has recently benefited from large-scale self-supervised learning (SSL) speech Transformers, serving as powerful feature extractors. However, existing systems rely on a static, scalar-based weighted fusion to handle multi-layer representations from SSL encoders, which suffer from biased layer u...
Jin Sob Kim, Hyun Joon Park, Wooseok Shin et al.· IEEE Signal Processing Lette...· 0 citations
Joint Acoustic Scene Classification (ASC) and Audio Tagging (AT) models traditionally rely on unconstrained multihead architectures. However, this design ignores the natural hierarchical dependency between sounds and acoustic scenes. In this work, we investigate a hierarchical (sequential) conditioning paradigm that pr...
Claudia Montero-Ramírez, Alba Martínez-Serrano, Carmen Peláez-Moreno· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.