Skip to content

CDCLAP: Efficient distillation framework for zero-shot audio event classification.

Oct 2026 · Neural Networks · Vol 206 Pt B, pp. 109764 · 0 citations · 37 references
Medicine

Abstract

This paper presents a novel Cross-Modal Distillation Contrastive Language-Audio Pretraining (CDCLAP) framework to address the challenges of zero-shot audio event classification (ZAEC) under limited hardware resources. The proposed method transfers visual knowledge from an ImageNet-pretrained MobileNetV3 to a lightweight student audio network, and employs a CLAP model-comprising HTSAT and natural language processing model GPT-2-as the teacher to enhance the representational capacity and semantic generalization of the audio encoder. Furthermore, a description generation strategy using GPT-2 is introduced to augment the audio-text pairs, thereby improving model performance. Extensive evaluations on three benchmark datasets-ESC50, US8K, and TUT17 demonstrate that CDCLAP achieves superior performance compared to existing state-of-the-art methods, while maintaining significantly lower model complexity. These results validate the effectiveness of the proposed cross-modal distillation strategy in improving zero-shot recognition accuracy and robustness, offering new insights into efficient multimodal audio understanding.

View source

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.