Skip to content
Preprint

Self-Supervised Object-Centric Representation for Multi-Task Audio Analysis

Oct 2026 · 0 citations · 17 references
Computer Science

TL;DR

Results demonstrate that slot-based representations capture transferable and compositional structure in audio scenes, enabling generalization across tasks and datasets.

Abstract

This paper studies the capability of an object-centric audio representation to support multiple downstream tasks within a unified framework. The object-centric model is trained in a self-supervised setup using slot attention by reconstructing Audio Spectrogram Transformer (AST) patch embeddings from a set of latent slots. It is trained on synthetic audio mixtures, which provides controlled multi-source audio scenes. These slot representations are evaluated for the classification, detection, and separation of sound events by training task specific output layers. On the classification task, the model achieves an F1-score of 0.8, comparable to clip-level AST embeddings which achieve 0.7. For sound event detection, it reaches a macro F1-score of 0.7, outperforming clip-level AST embeddings at 0.4 and matching AST patch embeddings at 0.7, which encode finer time-frequency information. In source separation, it achieves an SI-SDR improvement (SI-SDRi) of 18.4 dB. Furthermore, in a zero-shot source separation setting, the model achieves an SI-SDRi of 14.5 dB. These results demonstrate that slot-based representations capture transferable and compositional structure in audio scenes, enabling generalization across tasks and datasets.

View source

Similar papers

Open access 2026

CATF-Net: Content-Adaptive Multi-View Time–Frequency Fusion for Pretraining-Free Classification Across Heterogeneous Audio Tasks

Audio classification across speech-command and environmental-sound tasks requires both fine temporal cues and structured spectral context, yet many high-performing systems rely on external pretraining. We propose CATF-Net, a supervised framework trained from random initialization using only the labeled training data of...

Zhi-Cheng Pei, Jiang-Peng Li, Tong An et al. · 0 citations
#machine learning Preprint Oct 2026

LeAVJEPA: A Minimalist Architecture for Audio-Visual Self-Supervised Learning

Prior audio-visual self-supervised learning methods rely on mechanisms such as EMA target encoders, prediction heads, reconstruction decoders, and contrastive losses. We introduce LeAVJEPA, the first audio-visual encoder trained under LeJEPA's collapse-free objective. A single early-fusion Vision Transformer processes...

Benjamin Robson, Santeri Mentu, Wen-Shuai Zhao et al. · 0 citations
Oct 2026

CDCLAP: Efficient distillation framework for zero-shot audio event classification.

This paper presents a novel Cross-Modal Distillation Contrastive Language-Audio Pretraining (CDCLAP) framework to address the challenges of zero-shot audio event classification (ZAEC) under limited hardware resources. The proposed method transfers visual knowledge from an ImageNet-pretrained MobileNetV3 to a lightweigh...

Fei Chen, Hui Xue, Peng-Fei Fang · 0 citations
Open access 2026

Frame-Adaptive Layer Aggregation of Pre-Trained Speech Encoders for Speaker Verification

Speaker verification (SV) has recently benefited from large-scale self-supervised learning (SSL) speech Transformers, serving as powerful feature extractors. However, existing systems rely on a static, scalar-based weighted fusion to handle multi-layer representations from SSL encoders, which suffer from biased layer u...

Jin Sob Kim, Hyun Joon Park, Wooseok Shin et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Hierarchical Interpretable Multitask Audio Representation Learning: An Analysis of Event-Scene Relationships through the Information Bottleneck

Joint Acoustic Scene Classification (ASC) and Audio Tagging (AT) models traditionally rely on unconstrained multihead architectures. However, this design ignores the natural hierarchical dependency between sounds and acoustic scenes. In this work, we investigate a hierarchical (sequential) conditioning paradigm that pr...

Claudia Montero-Ramírez, Alba Martínez-Serrano, Carmen Peláez-Moreno · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.