Improving Cross-Modal Matching of College English Listening Materials Using Contrastive Learning

Abstract

Low audio-text matching accuracy in college English listening materials is caused by modality gaps, pronunciation variability, and scarce annotated data. To address this problem, this paper proposes a cross-modal alignment method based on contrastive learning. Wav2Vec 2.0 and BERT are used to extract speech and text features, respectively, followed by modality-specific projection networks that map heterogeneous representations into a shared semantic space. Under extreme data scarcity with a sample size of 5, semantically constructed positive and negative pairs and an in-batch hard negative mining strategy are introduced to improve discriminability. The InfoNCE loss optimizes cross-modal similarity, while a symmetric dual-tower encoder balances semantic consistency and feature independence. Experimental results show Top-1 accuracy of 94.7% ± 1.3% and Recall@5 of 97.6% ± 0.9%, outperforming AudioCLIP, CLAP, and Wav2CLIP baselines. A similarity gap of 0.58 at 10 dB confirms a clearly separated cross-modal decision boundary. The proposed approach mitigates modality disparity and small-sample limitations, providing a robust semantic alignment framework for intelligent listening instruction and low-resource multimodal signal matching.