MuseFlow-TSE, a discriminative deletion, generative insertion framework for music-bearing scenes, is presented, a music-aware cross-source band-split extractor that estimates target, interfering speakers, music, and noise under conservation and music-leakage constraints.
Abstract
Target speech extraction (TSE) isolates a speaker's voice given an enrollment utterance, but existing work is validated almost exclusively without music. Discriminative methods are content-faithful yet masking-bounded; generative methods improve naturalness but may hallucinate, drift in identity, and require many steps. We present MuseFlow-TSE, a discriminative deletion, generative insertion framework for music-bearing scenes. Stage-1 is a music-aware cross-source band-split extractor that estimates target, interfering speakers, music, and noise under conservation and music-leakage constraints. Stage-2 is a data-to-data flow refiner initialized by the discriminative estimate, negatively conditioned on interference stems, and run for 1--4 difficulty-adaptive Euler steps with an end-to-end fidelity objective.
Target speaker extraction (TSE) is a key frontend for enabling audio large language models (LLMs) to recognize and understand a designated speaker in overlapping speech. However, multilingual mixtures exhibit heterogeneous extraction demands, causing fixed-depth systems to either under-process challenging inputs or was...
Yan-Xiang Chen, Hao-Yuan Deng, Jin-Yu Wang et al.· 0 citations
A review of the current landscape and open problems in multimodal TSE from the perspective of auxiliary target cues drawn from multiple modalities and summarizes future directions concerning adaptive cue fusion, instruction-driven extraction, realistic evaluation, and trustworthy deployment.
Xin-Yuan Qian, Yang Zhou, Zi-Yang Jiang et al.· arXiv.org· 0 citations
Diarization-conditioned target-speaker ASR (TS-ASR), in particular Diarization-conditioned Whisper (DiCoW), is an accurate and data-efficient approach to multi-talker transcription. It has, however, three structural limitations: the model transcribes each speaker independently, operates on short (30 s) windows, and opt...
Alexander Polok, Dominik Klement, I. Medennikov et al.· 0 citations
Streaming audio-visual target speaker extraction (AV-TSE) uses a target speaker's face to extract the speaker's speech from an input audio mixture in real time. However, existing systems fix the target speaker for an entire clip, while conversation demands frequent and immediate target switches. To address this problem...
Hans-Sol Park, Min-Jing Song, Zinu Kim et al.· 0 citations
Deterministic target-speaker extraction (TSE) tends to preserve observed linguistic content but may extract the interfering speaker. In contrast, autoregressive generative TSE improves perceptual quality, but is prone to content hallucination and speaker identity drift, which can be mitigated by model grounding with th...
Han-Yu Meng, Hao Shi, Zhan-Hong He et al.· 0 citations
Self-supervised learning (SSL) is standard for speech representation learning, but mainstream models are designed around single-speaker audio, limiting their usefulness in multi-speakers scenarios. We present SepRQ, an open-source SSL framework that replaces masked prediction with a pseudo-source-separation objective o...
Séverin Baroudi, Hervé Bredin, R. Marxer· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.