It is suggested that many reported downstream gains reflect instance and seed dependent elicitation match, rather than universally improving the attainable performance ceiling, in self-supervised fine-tuning.
Abstract
Supervised fine-tuning (SFT) is widely used to adapt self-supervised speech representations to downstream classification tasks. Small gains observed under a single pretrained checkpoint are often interpreted as method-level improvements, i.e., a higher attainable performance ceiling. We show that such conclusions are not always reliable because SFT outcomes depend strongly on the specific pretrained instance. We conduct a systematic study on 3 SUPERB classification tasks, evaluating 8 SFT variants across 9 pretrained checkpoints from wav2vec~2.0, HuBERT, and WavLM, with multi-seed repetitions on representative base-scale models. We find that the identity of the statistically indistinguishable top-group SFT recipe is often checkpoint-dependent, with limited transferability across pretrained instances. These findings suggest that many reported downstream gains reflect instance and seed dependent elicitation match, rather than universally improving the attainable performance ceiling.
A controlled experiment on the final window of pretraining, the last data trained on before instruction tuning, finds that what a model is pretrained on last shapes how it reacts to alignment, and what it was trained on last should be reported with it.
Cen Lu, Yung-Chen Tang, Andrea Cavallaro· 0 citations
BEST-RQ-2 is presented, an evolution of BEST-RQ that retains frozen randomprojection-based discrete targets while introducing a two-step contextualize-then-predict pretraining scheme, and consistently outperforms one-stage baselines in overall transfer while keeping inference compute unchanged.
Ludovic Tuncay, Etienne Labbé, Thomas Pellegrini· arXiv.org· 0 citations
LP-SFT, a Local-Preserving Supervised Fine-Tuning objective designed to explicitly protect this inherent entropy structure, improves overall performance over vanilla SFT and recent SFT-enhancement baselines, suggesting that local preservation helps mitigate capability degradation without collapsing sampling-accessible diversity.
Yueyang Wang, Baolong Bi, Shuo Lu et al.· 0 citations
When fine-tuning large language models, the assumption is that more data means better results. This principle is often extended to text-to-speech (TTS) fine-tuning, yet remains underexplored, particularly for low-resource languages where high-quality data is difficult to obtain. In this work, the "more data is better" phenomenon is investigated for Turkish TTS using XTTS v2 by incrementally increasing training data. Speech quality is evaluated using multiple metrics, including UTMOS, NISQA, and an LLM-based TTS evaluation framework. For the LLM-based evaluation, a multimodal language model (Gemini) was prompted to assess Turkish-specific pronunciation, naturalness, and synthesis artifacts on a calibrated 1-10 scale. Results suggest that for TTS fine-tuning on non-mainstream languages, modest data investments may achieve near-optimal quality.
Doruk Berke Yurtsizoğlu, Ismail Karakaya, Serdar Kalaycı et al.· Signal Processing and Commun...· 0 citations
This paper describes TalTech's submissions to the Beyond Transcription Challenge (BeTraC), which requires generating SOAP notes directly from long doctor-patient conversation recordings, without intermediate transcription. After screening open-weight speech LLMs for long-audio robustness, we adapted Voxtral Mini (lightweight track) and Voxtral Small (heavyweight track) with LoRA supervised fine-tuning followed by DAPO reinforcement learning that uses the challenge metric, Open Medical Concept F1, as its reward. Our systems ranked first in both tracks, and an independent LLM-as-a-judge evaluation showed the lowest hallucination rate among all submissions, indicating that reinforcement learning against a concept-matching metric need not compromise factual reliability. We also find that fine-tuning on text transcripts transfers well to speech input and appears to improve robustness on out-of-domain real recordings.
The results show that a small, carefully chosen activation set can switch on test-time scaling inside a frozen ASR model without corrupting the model itself, converting fixed per-utterance compute into input-dependent compute where it is most needed.
Ho-Lam Chung, Yiming Chen, Dau-Cheng Lyu et al.· 0 citations