Speak in Any Language: An End-to-End Framework for Faithful Multilingual Dubbing

Abstract

The growing demand for multilingual digital content has increased the need for automated video dubbing systems capable of producing linguistically accurate and temporally synchronized speech while preserving speaker identity. Existing approaches often address transcription, translation, and speech synthesis independently, leading to synchronization errors and inconsistent speaker representation in the final dubbed output. This paper presents an end-to-end multilingual video dubbing framework that introduces a duration-aware translation strategy to preserve temporal alignment between translated speech and the original video, together with a speaker-aware hybrid textto-speech architecture that adaptively selects suitable synthesis models to improve naturalness and scalability across multiple languages. The framework integrates speech enhancement, wordlevel transcription and alignment, speaker diarization, neural machine translation, adaptive speech synthesis, subtitle generation, and synchronized video reconstruction into a unified pipeline. Experimental evaluation demonstrates competitive transcription accuracy with a WER of 6.34% and CER of 2.47%, while achieving strong semantic preservation measured using BLEU and BERTScore, high speech naturalness evaluated using MOS or DNSMOS, and improved temporal synchronization with significantly reduced duration alignment error across diverse target languages. The results demonstrate that the proposed framework generates semantically faithful, naturally sounding, and temporally synchronized multilingual dubbed videos while maintaining speaker consistency, providing a scalable solution for multilingual content localization.