Self-supervised audio representation learning model based on time-frequency decoupling and masked reconstruction

Abstract

Self-supervised learning (SSL) provides an effective way to obtain transferable representations from unlabeled audio. Existing spectrogram-based teacher-student methods mainly emphasize cross-view consistency and insufficiently model the distinct temporal/frequency structure and local acoustic information. This paper proposes a self-supervised model that integrates a time-frequency decoupled (TF-D) stem with masked latent reconstruction. The stem separately refines temporal variations and spectral correlations before patch embedding, while the masked branch predicts continuous, stop-gradient teacher features to complement global predictive self-distillation with a local constraint. The model is pre-trained from scratch on AudioSet-20K and evaluated on five downstream tasks using frozen encoders. It achieves 0.336 mean average precision (mAP) on AudioSet-20K, a 20.4% relative improvement over the internal ATST-S baseline, and accuracies of 84.48%, 98.1%, 74.1%, and 89.7% on UrbanSound8K, Speech Commands V2, VoxCeleb1, and NSynth, respectively. The full model gives the strongest overall transfer results among the internal variants and a competitive balance among representation quality, encoder scale, and inference efficiency.