It is suggested that many reported downstream gains reflect instance and seed dependent elicitation match, rather than universally improving the attainable performance ceiling, in self-supervised fine-tuning.
Abstract
Supervised fine-tuning (SFT) is widely used to adapt self-supervised speech representations to downstream classification tasks. Small gains observed under a single pretrained checkpoint are often interpreted as method-level improvements, i.e., a higher attainable performance ceiling. We show that such conclusions are not always reliable because SFT outcomes depend strongly on the specific pretrained instance. We conduct a systematic study on 3 SUPERB classification tasks, evaluating 8 SFT variants across 9 pretrained checkpoints from wav2vec~2.0, HuBERT, and WavLM, with multi-seed repetitions on representative base-scale models. We find that the identity of the statistically indistinguishable top-group SFT recipe is often checkpoint-dependent, with limited transferability across pretrained instances. These findings suggest that many reported downstream gains reflect instance and seed dependent elicitation match, rather than universally improving the attainable performance ceiling.
A controlled experiment on the final window of pretraining, the last data trained on before instruction tuning, finds that what a model is pretrained on last shapes how it reacts to alignment, and what it was trained on last should be reported with it.
Lujing Cen, Yung-Chen Tang, Andrea Cavallaro· arXiv.org· 0 citations
BEST-RQ-2 is presented, an evolution of BEST-RQ that retains frozen randomprojection-based discrete targets while introducing a two-step contextualize-then-predict pretraining scheme, and consistently outperforms one-stage baselines in overall transfer while keeping inference compute unchanged.
Ludovic Tuncay, Etienne Labbé, Thomas Pellegrini· arXiv.org· 0 citations
LP-SFT, a Local-Preserving Supervised Fine-Tuning objective designed to explicitly protect this inherent entropy structure, improves overall performance over vanilla SFT and recent SFT-enhancement baselines, suggesting that local preservation helps mitigate capability degradation without collapsing sampling-accessible...
Yueyang Wang, Baolong Bi, Shuo Lu et al.· 0 citations
When fine-tuning large language models, the assumption is that more data means better results. This principle is often extended to text-to-speech (TTS) fine-tuning, yet remains underexplored, particularly for low-resource languages where high-quality data is difficult to obtain. In this work, the "more data is better"...
Doruk Berke Yurtsizoğlu, Ismail Karakaya, Serdar Kalaycı et al.· Signal Processing and Commun...· 0 citations
This paper describes TalTech's submissions to the Beyond Transcription Challenge (BeTraC), which requires generating SOAP notes directly from long doctor-patient conversation recordings, without intermediate transcription. After screening open-weight speech LLMs for long-audio robustness, we adapted Voxtral Mini (light...
The results show that a small, carefully chosen activation set can switch on test-time scaling inside a frozen ASR model without corrupting the model itself, converting fixed per-utterance compute into input-dependent compute where it is most needed.
Ho-Lam Chung, Yiming Chen, Dau-Cheng Lyu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
