Exploring Wav2Vec2 XLSR-53 and Whisper-small for Telugu language speech-to-text: a fine-tuning approach

Abstract

Online voice-based applications and speech communication have grown as a result of the revolutionary rise of smart gadgets and social media. The rapid advancement of deep learning (DL) has transformed the field of audio processing, enabling smooth human-computer interaction. DL approaches have been used to develop speech-to-text (STT) systems across various languages and topics. These models require a large amount of training data: extensive corpora of continuous speech utterances collected from numerous speakers, along with their corresponding transcripts. In this study, we explore the use of state-of-the-art pre-trained models like Wav2Vec2 XLSR-53 and Whisper-small for developing STT systems in the Telugu language, addressing the challenge of limited data availability and demonstrating satisfactory results [23.62% Word Error Rate (WER), 4.12% Character Error Rate (CER)] even when fine-tuned on a smaller dataset. To evaluate model performance, we employed a k-fold cross-validation approach with values of k = 2 to k = 5, and compared the results with the conventional train-test split method. The results indicate that at k = 5, the Wav2Vec2 XLSR-53 model achieved a cross-validation Character Error Rate(WER) of 23.62% and a Character Error Rate (CER) of 4.12%, and the Whisper-small model yielded a cross-validation WER of 28.67% and a CER of 5.48%. These findings suggest that the k-fold cross-validation strategy, at k = 5, enhances the robustness of Wav2Vec2 XLSR-53 in low-resource language scenarios where training data is limited. For Whisper-small, however, the baseline train-test split (WER: 27.76%) outperformed all k-fold configurations tested (k = 2: 32.05%, k = 3: 29.72%, k = 4: 29.40%, k = 5: 28.67%), indicating that the benefit of k-fold cross-validation observed for Wav2Vec2 XLSR-53 does not generalize across model architectures. Additionally, the models were evaluated on unseen noisy data, and both models demonstrated satisfactory performance.