This paper investigates how multilingual medical adaptation reshapes the internal representations of Whisper models through layer-wise encoder analysis, and shows that English medical fine-tuning produces the dominant encoder shift, whereas multilingual continuation largely preserves the adapted representation space.
Abstract
Medical automatic speech recognition (MedASR) requires adaptation to specialised terminology, limited annotated clinical data, and multilingual use cases. Although large-scale pretrained ASR models such as Whisper achieve strong generalisation, their behaviour after medical and multilingual adaptation remains insufficiently understood beyond word error rate (WER). This paper investigates how multilingual medical adaptation reshapes the internal representations of Whisper models through layer-wise encoder analysis. We compare zero-shot decoding, English-only fine-tuning, German-only diagnostic fine-tuning, two-stage EN->EN+DE continuation, and direct EN+DE fine-tuning across Whisper model sizes. Fine-tuning substantially improves MedASR performance, but the best model depends on the adaptation setting: Whisper-Medium gives the lowest English WER (7.72%) and the lowest combined EN+DE WER under direct EN+DE training (26.30%); German-only Whisper-Large-v3 gives the lowest German WER (44.96%), but as a within-corpus diagnostic on 86 single-speaker training utterances rather than robust generalisation. Layer-wise analysis of the two-stage Whisper-Small trajectory shows that English medical fine-tuning produces the dominant encoder shift, whereas multilingual continuation largely preserves the adapted representation space. Domain and language information remain highly recoverable across layers, while linearly recoverable error-predictive cues weaken as WER improves.
Although Whisper models benefit from large-scale multilingual pre-training, their performance on Burmese medical speech remains limited. This work presents a Burmese medical speech recognition framework built on a high-quality 28-hour corpus recorded and validated by native speakers. We fine-tune Whisper models using full fine-tuning (FFT) and parameter-efficient fine-tuning (PEFT) with LoRA. To evaluate robustness, we apply waveform- and spectrogram-level data augmentation under controlled noise and simulated room acoustics. While augmentation reduces performance on clean speech, it significantly improves robustness in noisy and reverberant environments across FFT and PEFT settings. Our best-performing system, fully fine-tuned myMediWhisper-Medium without augmentation, achieves a state-of-the-art Word Error Rate (WER) of 23.44%, outperforming much larger general-domain fine-tuned models. Dataset and other resources can be found at the Huggingface repository: https://huggingface.co/datasets/LULab/mediTalk-mm-rdy.
Ye Kyaw Thu, Y. Lin, Thura Aung et al.· 0 citations
Developing Automatic Speech Recognition (ASR) systems for specialized medical domains is challenging in lowresource settings due to the limited availability of annotated speech data. Synthetic speech generated using text to speech (TTS) systems is often used to augment training data, but directly mixing synthetic and real speech can introduce distribution mismatch that degrades encoder representations. This work introduces Real-Governed Representation-Calibrated Training (RG-RCT), a training strategy that regulates the influence of synthetic speech at the representation level. The proposed framework combines reliability-based confidence estimation, representation alignment, and layer-wise representation governance to stabilize the encoder feature space while preserving acoustic diversity from multi-speaker synthetic data. Experiments on medical-domain speech datasets using Wav2Vec2 and Whisper demonstrate that RG-RCT consistently outperforms conventional strategies such as direct mixing and confidence-aware training. The proposed method achieves the lowest Word Error Rate (WER), reaching 17.1% and 16.6% for Telugu and 23.3% and 25.6% for Kannada. These results indicate that regulating encoder representation geometry enables more effective utilization of synthetic speech for low-resource medical ASR.
S. Devarakonda, Ravi Sastry Kolluru, Srihari Bandarupalli et al.· International Conference on...· 0 citations
BACKGROUND: Artificial intelligence (AI) medical scribes rely on speech-to-text (STT) models for transcription. Evaluations of STT models in non-English settings remain scarce. We benchmarked ten STT models on medical consultations from Latin American (LatAm) Spanish and assessed whether fine-tuning improves transcription accuracy. METHODS: Ten YouTube videos depicting medical consultations. Human transcriptions were the ground truth. Five open-source models were evaluated: Whisper Large, Whisper Large v3, Whisper Large v3 Turbo, Voxtral Mini 3B, and Canary 1B v2; and so were five close-source models: gpt-4o-transcribe, gpt-4o-mini-transcribe, gemini-2.5-pro, Eleven Labs, and Assembly AI. Whisper Large v3 was fine-tuned. One video was withheld from training. Performance assessed using Word Error Rate (WER), Character Error Rate (CER), BLEU Score, ROUGE-L, BERT Score, and Semantic Similarity on the one withheld video. RESULTS: None of the fine-tuning iterations outperformed the vanilla Whisper Large v3. With the withheld video, Gemini-2.5-pro was the close-source model with the best performance in four of six metrics. In comparison to the close-source models, the fine-tuned model never outperformed the other models (withheld video); conversely, in comparison to the close-source models, the fine-tuned model showed better performance across metrics, for instance: BLEU score (63% vs to 58% for the second-ranking model), BERT (89% vs to 86%), and semantic similarity (89% vs to 83%), CER (19% vs 20%). CONCLUSIONS: Whisper Large v3 and its fine-tuned variant are the best open-source STT models for transcribing medical conversations in LatAm Spanish. These findings provide an evidence base for developing AI medical scribes tailored to Spanish-speaking LatAm.
R. M. Carrillo, A. Carbajal Serrano, P. S. Condori Pinedo· medRxiv· 0 citations
Accurate clinical speech recognition remains challenging because rapid pronunciation, domain-specific terminology, and background noise often degrade automatic speech recognition and subsequent medical record generation. This study proposes a multi-stage intelligent documentation framework that integrates a 12-layer Conformer architecture, BERT-BiLSTM-CRF semantic modeling, and BART-based structured text generation. The Conformer encoder captures both local acoustic characteristics and long-range contextual dependencies, while the semantic module performs medical entity recognition and normalization to enhance terminology consistency. The extracted information is subsequently incorporated into a BART generator with clinical knowledge prompts to produce standardized SOAP-compliant medical records. Experimental results demonstrate a word error rate of 6.3%, medical term accuracy of 95.8%, low response latency of approximately 940–960 ms, and generation quality approaching physician-written records. Beyond clinical documentation, the proposed framework illustrates the effectiveness of deep time-frequency feature extraction and contextual sequence modeling for complex noisy signals, offering methodological insights for electromagnetic signal interpretation, antenna measurement data processing, and intelligent information extraction in propagation-related applications.
X. Zou, L. Wang, J. Sun et al.· Advanced Electromagnetics· 0 citations
Large Language Models (LLMs) perform strongly in English medical tasks but degrade substantially in Arabic, a gap widely attributed to limited training data. We systematically investigate this assumption via tuned lens probing and causal activation patching, and find that Arabic medical knowledge is present in intermediate model representations but fails to surface at the output. This mechanistic insight motivates a targeted adaptation strategy: rather than fine-tuning the full network, we propose Targeted Low-Rank Adaptation (TLoRA), restricted to the layer window where cross-lingual representations diverge, upstream of the output layers where the failure manifests. We evaluate TLoRA on multiple-choice medical QA, where our approach outperforms full-network LoRA, zero-shot, and few-shot baselines. We further evaluate it on short-answer generation and multi-turn clinical dialogue, where it performs competitively without the need for task-specific finetuning. We additionally introduce AraClinicDialog, a clinician-constructed Arabic medical dialogue benchmark in MSA with validated variants across four Arabic dialects. Together, these contributions demonstrate that mechanistic diagnosis can serve as a practical guide for targeted adaptation in underrepresented-language medical LLMs.
Chaimae Abouzahir, Musa Khan, Hala Ali-Hassan et al.· 0 citations
This study investigates the English-to-Turkish clinical translation performance of three large language models (Aya-23, Seed-X-Instruct, Gemma-3) using zero/few-shot prompting and QLoRA fine-tuning. Our three-stage setup evaluates: (1) general translation after FLORES-devtest training, (2) medical domain adaptation on i2b2 via a mixed dataset, and (3) generalization capacity on the independent FLORES-test set after mixed-data fine-tuning, using SacreBLEU, chrF, and COMET. Results show that Seed-X-Instruct achieves the strongest fine-tuned performance, reaching 32.70 BLEU, 64.34 chrF, and 90.65 COMET on the general FLORES test set, and 22.40 BLEU, 54.61 chrF, and 79.70 COMET on the medical i2b2 set. After mixed-data training, Seed-X improves from 11.48 to 32.39 BLEU on the unseen FLORES test set, whereas Aya-23 drops from 19.23 to 10.83 and Gemma-3 largely preserves its strong zero-shot baseline. These findings indicate that QLoRA effects are model-dependent and that domain adaptation does not yield the same generalization behavior across models.
Muhammetalp Erdem, Ş. Bora, Oguz Dikenelli· Signal Processing and Commun...· 0 citations
A new method for surgically removing training examples from a model reveals that as datasets grow, the link between what a model learns and what it produces dissolves.
