Disentangled Speaker Flow with Time-Decayed Conditioning for Dialogue Text-to-Speech

Abstract

Dialogue synthesis requires modeling speaker transitions while maintaining high linguistic accuracy. Existing text-to-speech frameworks face an inherent trade-off between computational efficiency and controllability, leading to either high implementation cost or limited flexibility in speaker transitions. In this work, building upon ZipVoice-Dialog, we propose a speaker modeling strategy that disentangles the speaker-turn condition from linguistic representations in the flow matching process. We also introduce a time-dependent decay mechanism that progressively reduces the strength of speaker-turn condition along the trajectory, allowing early steps to preserve speaker traits while later steps emphasize linguistic refinement. This dynamic modulation mitigates speaker content interference and stabilizes cross turn transitions without increasing model complexity. Experiments demonstrate improved controllability and competitive performance, along with strong generalization capability.