LoRA-MoE, a parameter-efficient adaptation method that combines Low-Rank Adaptation with a mixture of experts (MoE) to improve speech recognition for individuals with Parkinson’s disease (PD), demonstrates consistent improvements and stable performance across all severity levels, and its performance is robust to the number of experts.
Abstract
Despite advances in automatic speech recognition (ASR) technology, its performance remains limited when processing dysarthric speech, which hinders people with dysarthria from accessing voice-based digital technologies. In this study, we propose LoRA-MoE, a parameter-efficient adaptation method that combines Low-Rank Adaptation (LoRA) with a mixture of experts (MoE), to improve speech recognition for individuals with Parkinson’s disease (PD), who often exhibit hypokinetic dysarthria. We integrate LoRA-MoE into the Whisper model by freezing its pre-trained weights while introducing multiple LoRA experts and a router network that dynamically combines them based on the input. Experiments on a Korean PD speech dataset show that LoRA-MoE achieves a character error rate (CER) of 12.01%, a 38.54% relative improvement over the baseline Whisper model, and outperforms full fine-tuning and single LoRA adaptation by 46.00% and 11.37%, respectively, with consistent improvements across all PD severity levels. The proposed approach demonstrates consistent improvements and stable performance across all severity levels, and its performance is robust to the number of experts; we adopt three experts as our default configuration.
A layer-wise analysis of nine SSL speech backbones using a low-capacity logistic regression probe reveals that the transferred discriminative signal lacks pathological specificity, highlighting critical limitations that must be addressed before speech-based pathology recognition models can be reliably deployed in clinical settings.
S. Kopar, Sam Gijsen, Abner Hernandez et al.· 0 citations
Per-patient adapters are the preferred production architecture for dysarthric automatic speech recognition (ASR), yet parameter-efficient fine-tuning (PEFT) variants have not been compared in the speaker-dependent, per-patient regime. We present a single-speaker case study comparing seven LoRA-family methods (LoRA, QLoRA, AdaLoRA, DoRA, LoHA, VeRA, VB-LoRA) on two production bases (Whisper-large-v3 with Hungarian fine-tuning, and a multilingual Qwen3-ASR-1.7B checkpoint) for one post-stroke Hungarian male speaker (S1, 409 utterances; severe dysarthria on auditory-perceptual clinical assessment). Attention-projection adapters substantially improve CER on both bases. Across three seeds, a paired bootstrap detects no significant LoRA-DoRA difference (p>0.5; 13.86/13.90 % CER on Whisper, 28.10/28.33 % on Qwen3-ASR), so we adopt the simpler, cheaper LoRA. Real 4-bit (NF4) QLoRA is worse on every seed and both bases (14.56/30.09 % CER) with no memory saving at this scale, and LoHA, VeRA, VB-LoRA and AdaLoRA do not reach the LoRA family, though LoHA still gives an 18.6 % relative CER reduction on Whisper. On the same base, full fine-tuning is more accurate (11.43 % CER), but a 115 MB LoRA that also adapts the feed-forward blocks reaches within 0.66 pp of it at approximately 3.7 % of the per-patient storage. A 6-point enrollment grid shows about 5 min of patient audio captures 45.6 % of the zero-shot-to-30-min CER reduction, with further gains at 10 and 30 min (caveat: one speaker, one language, severe post-stroke dysarthria). Training scripts and recipes will be released, source-available under a research-use licence, on publication.
Speech enhancement models optimised for perceptual quality metrics such as PESQ do not necessarily improve automatic speech recognition (ASR) performance, as aggressive noise suppression can distort the phonemic cues that acoustic models rely on. We propose a time-domain speech frontend based on the Mamba selective state space model, trained exclusively with ASR-oriented loss functions to preserve phonetically discriminative structure rather than maximise perceptual quality. The model follows a U-Net encoderdecoder architecture with 4 × temporal downsampling and six stacked bidirectional Mamba blocks operating on the raw waveform, trained with a multi-component loss combining multiresolution spectral supervision, log-mel feature matching, speech correlation preservation, and ideal-ratio-mask spectral objectives. On VoiceBank-DEMAND, the proposed model achieves 10.26% WER with wav2vec2-base-960h and 7.68% with whisper-base, achieving statistically equivalent WER to SE-Mamba (10.30% / 7.73%) while producing a lower PESQ of 3.10 versus 3.69. Phoneme-class analysis confirms that the proposed model matches or outperforms SEMamba on five of seven articulatory classes, demonstrating that ASR-oriented training objectives achieve equivalent robustness to perceptual enhancement models without modifying the downstream ASR system.
Reshma K R, A. Vuppala· International Conference on...· 0 citations
Aphasia affects a person's ability to retrieve words and coordinate the movements of the speech articulators, resulting in impaired speech production and pronunciation. Current word prediction tools do not use lip movement patterns, which limits their usefulness for people with speech disorders. This study aims to predict the word a person intends to say by analyzing both lexical activation and articulatory stability. A Multi-Frame Transformer (MFT) model that processes five consecutive frames of lip movements is developed, along with a proposed Single-Frame Transformer (SFT) and three pre-trained comparative models including BERT, RoBERTa, and a 3D CNN. The proposed MFT model achieved an 84.50% accuracy, outperforming BERT with 63.33%, the 3D CNN with 51.11%, the Single-Frame Transformer with 26.67%, and RoBERTa with11.11%. The proposed model maintained 74% accuracy when word-finding information was missing and 58% accuracy when lip movement information was missing. These findings show that motor signals are more critical for correct prediction. The Multi-Frame Transformer (MFT) model can effectively predict intended words from lip movement patterns, offering a foundation for future communication aids for people with aphasia.
N. S. A. Azhar, Nik Mohd Zarifie Hashim, M. N. Mohd et al.· NED University Journal of Re...· 0 citations
Fine-tuning of large pre-trained models, such as Whisper, has gained prominence in the area of speech processing. Since full fine-tuning requires a large amount of data as well as high-ended computational resources, parameter efficient fine-tuning (PEFT) has been the preferred choice among researchers. During the past few years, several PEFT techniques have been developed and have been observed to be extremely effective. Motivated by the success of PEFT, in this paper, we have investigated and documented the efficacy of Low-Rank Hadamard Product Adaptation (LoHA) of Whisper models for children's automatic speech recognition (ASR) task especially in limited data scenario. We have also compared LoHA with a few other PEFT variants. Even though LoHA has been explored for signal processing and federated learning tasks, its impact on children's ASR has not yet been studied. Experimental results presented in this paper indicate that applying LoHA significantly influences ASR performance, achieving a word error rate of 3.0%.
st Ankita, S. Shahnawazuddin· International Conference on...· 0 citations
CASA, a simpler architecture combining Whisper-medium and Qwen3.5-2B that achieves state-of-the-art performance while providing a more interpretable separation between speech delivery and content, is proposed.
Nhan Phan, Ilona Lähteenmäki, Anna von Zansen et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
