RG-RCT: Real-Governed Representation-Calibrated Training for Low-Resource Medical ASR

Abstract

Developing Automatic Speech Recognition (ASR) systems for specialized medical domains is challenging in lowresource settings due to the limited availability of annotated speech data. Synthetic speech generated using text to speech (TTS) systems is often used to augment training data, but directly mixing synthetic and real speech can introduce distribution mismatch that degrades encoder representations. This work introduces Real-Governed Representation-Calibrated Training (RG-RCT), a training strategy that regulates the influence of synthetic speech at the representation level. The proposed framework combines reliability-based confidence estimation, representation alignment, and layer-wise representation governance to stabilize the encoder feature space while preserving acoustic diversity from multi-speaker synthetic data. Experiments on medical-domain speech datasets using Wav2Vec2 and Whisper demonstrate that RG-RCT consistently outperforms conventional strategies such as direct mixing and confidence-aware training. The proposed method achieves the lowest Word Error Rate (WER), reaching 17.1% and 16.6% for Telugu and 23.3% and 25.6% for Kannada. These results indicate that regulating encoder representation geometry enables more effective utilization of synthetic speech for low-resource medical ASR.