Skip to content
Preprint

Listen, Critique, and Refine: RL-Based Self-Refinement for Instruction-Following Speech Synthesis

Sep 2026 · 1 citation · 35 references
Engineering Computer Science

TL;DR

This work training a LALM with reinforcement learning to reason over its own speech output generates a draft speech as a form of audio-token reasoning, critiques its own generation by reflecting on the acoustic realization in text, and produces a refined version conditioned on both the first-pass speech and the critique.

Abstract

Large Audio Language Models (LALMs) can follow diverse instructions to synthesize speech in specified styles. However, complex instructions that require simultaneous control over pitch dynamics, speaking rate, and emotional tone often exceed what a single-pass generation can faithfully realize. While recent reasoning models have shown that intermediate"thinking"tokens improve output quality, this paradigm has been confined to the text modality. In this work, we extend reasoning to the audio token space by training a LALM with reinforcement learning to reason over its own speech output. The model first generates a draft speech as a form of audio-token reasoning, critiques its own generation by reflecting on the acoustic realization in text, and then produces a refined version conditioned on both the first-pass speech and the critique, all within a single model. After RL training, the refined two-hop outputs achieve a relative improvement of 7.15\% on the InstructTTSEval benchmark, demonstrating the model's reflective ability.

View source

Similar papers

Voice of Reason: Reinforcement Learning for Spoken Math

This work first adapts the model to the domain using supervised fine-tuning on synthesized spoken question-answering data and shows that RL improves the accuracy on GSM8K beyond levels previously achieved for speech models only with supplementary reasoning traces, and establishes a new state-of-the-art for mathematical...

Timothée Weisselberger, Edouard Graves, Alexandre Défossez · 0 citations
Preprint Sep 2026

Post-Training Zero-Shot TTS for Fine-Grained Emotion and Duration Control via Natural Language

A unified post-training framework that equips pretrained text-to-speech models with natural-language control over segment-level emotion and duration is proposed, highlighting post-training as a practical approach to extending existing speech synthesis models.

Lian-Ru Gao, Yu-Jie Guo, Yong Qin · 0 citations
#machine learning Preprint Oct 2026

AdaLoop: Adaptive-Depth Latent Reasoning for Audio Language Models

Large audio language models answer questions about speech, sound, and music, yet their accuracy drops sharply on tasks that need fine-grained acoustic analysis. Judging which of two speakers has the higher pitch demands iterative signal-level reasoning that a content question does not. Current models spend the same com...

Seung-Woo Lee, Bo-Wen Qi · 0 citations
Preprint Aug 2026

SpeechGym: An Audio-Native Gym for Training Voice Agents via Reinforcement Learning

SpeechGym is presented, an audio-native agentic environment in which two omni-modal models converse in native audio, with no external ASR or TTS and no API boundary, over the unmodified tasks, tools and success check of an established text agentic benchmark, so that the interaction modality is the only variable and the...

Jia-Jun Fan, Jing-Yuan Li, Prashanth Gurunath Shivakumar et al. · 1 citation · ⚡1
#natural language process... Preprint Oct 2026

Beyond Speech Captions: Speech-Rewarded Style Planning for Conversational Text-to-Speech

Speech-Rewarded Style Planning (SRSP), which trains a text-based style planner through a frozen downstream TTS model, achieves higher speech-style and emotion similarity to target speech and lower mel-cepstral distortion than the Base LLM and target-audio-informed captioning baselines.

Shi-Yue Zhu, Lian-Bo Liu, Sizhen Lyu et al. · 0 citations
Preprint Sep 2026

EmphTTS: an emphasis-control TTS with reinforcement learning

Generating controllable and human-like emphasis remains an open challenge in text-to-speech, even when explicit emphasis control signals are provided in the text input, limiting the communicative accuracy of synthetic speech in real-world applications. Reinforcement learning has recently shown promise for post-training...

Zirui Li, Rech Silas, Lauri Juvela et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.