Preprint

Iterative Self-Learning for Expressive Text-to-Speech Synthesis

Nicholas Sanders G. Henter Simon King Korin Richmond

Aug 2026 · 0 citations · 55 references

Engineering Computer Science

TL;DR

In the most data-scarce conditions, ISL-trained models outperform single-pass pseudo-labeling and further approach fully supervised performance, demonstrating that gradient-based ISL is an effective solution to expressive label scarcity in low-resource TTS.

Abstract

Expressive text-to-speech (TTS) systems that use explicit conditioning labels provide direct and interpretable control over expressive attributes, in contrast to reference-based or prompting-based approaches, but require labeled data. Obtaining these labels at scale is costly and time-consuming, yet no prior semi-supervised framework addresses this specific bottleneck. Existing semi-supervised TTS methods instead target scarcity of paired speech-text data or transcriptions. To address the scarcity of expressive labels, we propose an Iterative Self-Learning (ISL) framework for expressive TTS, built on Invert-Classify, a classifier-free method that recovers discrete expressive labels by inverting a frozen generative model. The framework iteratively pseudo-labels unlabeled speech using the current model, retrains on the combined labeled and pseudo-labeled data, and repeats, progressively refining label quality and synthesis. We validate on two expressive tasks, word-level prominence and utterance-level emotion, across multiple low-resource data splits. We find that iterative refinement can improve pseudo-label accuracy over single-pass baselines. Furthermore, we observe that these improvements in pseudo-labeling of expressivity translate to gains in expressive label adherence and synthesis quality, confirmed by objective metrics and human listening tests. In the most data-scarce conditions, ISL-trained models outperform single-pass pseudo-labeling and further approach fully supervised performance, demonstrating that gradient-based ISL is an effective solution to expressive label scarcity in low-resource TTS.

arXiv Semantic Scholar

View source

Similar papers

Preprint Sep 2026

Post-Training Zero-Shot TTS for Fine-Grained Emotion and Duration Control via Natural Language

Audiobook narration, conversational agents, and audiovisual dubbing require speech that conveys changing emotions and adapts its pacing within a single utterance. But most existing TTS systems typically rely on utterance-level style conditioning, making such fine-grained control difficult to achieve. In light of this, and inspired by the success of post-training in large language models, we propose a unified post-training framework that equips pretrained text-to-speech models with natural-language control over segment-level emotion and duration. Supervised fine-tuning establishes instruction-conditioned speech generation, while reinforcement learning with group relative policy optimization refines control accuracy using emotion and duration rewards alongside content and speaker preservation objectives. By reusing the pretrained architecture, our approach avoids additional inference-time control modules. Experiments demonstrate significantly improved fine-grained controllability while maintaining speech intelligibility and speaker identity, highlighting post-training as a practical approach to extending existing speech synthesis models.

Lianru Gao, Yujie Guo, Yong Qin · 0 citations

Open access Aug 2026

Detecting Self-Repairs from Spontaneous Speech with Prompt Ablation Across LLMs and Fine-Tuned Encoder

This work compares the capability of generative LLMs under a five-condition prompt ablation against a fine-tuned DistilBERT token classifier at detecting self-repairs and suggests that a locally deployable encoder, given sufficient in-domain annotation, is a more plausible route to clinical self-repair detection than scaling model size or prompt complexity.

R. Wu, S. Pugh, K. O'Connor et al. · medRxiv · 0 citations

Models are Zero-Shot Text

Experimental results show that VALL-E outperforms the state-of-the-art zero-shot TTS system in terms of speech naturalness and speaker similarity and could preserve the speaker’s emotion and acoustic environment from the prompt in synthesis.

Unknown authors · 0 citations

Preprint Aug 2026

CuteTTS: Efficient and High-Quality Speech Synthesis via Autoregressive Modeling of Continuous Latents

CuteTTS is presented, a compact continuous-autoregressive TTS system that reconciles high-fidelity generation with the latency demands of real-time interaction and introduces guidance-step distillation, which absorbs classifier-free guidance and multiple solver steps into a single interval-conditioned student.

Yuqian Zhang, Yao Shi, Kexin Huang et al. · 0 citations

Preprint Aug 2026

Towards More Expressive Spoken LLMs: Fine-Grained Intent Benchmarking and Acoustic-Lexical Decoupled Policy Optimization

ParaIntent is introduced, a Chinese benchmark comprising 14 intent categories with balanced explicit and implicit samples, together with a multidimensional evaluation protocol covering intent fulfillment, response quality, and emotional expression, which improves over standard GRPO on most automatic metrics and achieves the best subjective results among the fine-tuned variants.

Xiangning Lin, Tian-Hao Zhang, Chunfeng Wang et al. · 0 citations

Preprint Aug 2026

Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder

A multilingual zero-shot TTS system that supports 14 languages and performs both intra-lingual and cross-lingual reference cloning without requiring transcripts of audio prompts, which achieves the best average overall rank in human evaluation among recent open-source and commercial systems.

Hua-Xuan Wang, Hui-Min Wang, Ruiyu Zhang et al. · 0 citations