REAL-2MIX, a Mandarin AV-TSE benchmark of jointly recorded real two-speaker mixtures with synchronized multi-view video, and REAL-2MIX-AVSE, a Mandarin AV-TSE benchmark of jointly recorded real two-speaker mixtures with synchronized multi-view video are introduced.
Abstract
Audio-visual target speaker extraction should return the speaker indicated by the video, yet a separator can ignore the visual cue and repeatedly output the acoustically dominant voice. We introduce REAL-2MIX, a Mandarin AV-TSE benchmark of jointly recorded real two-speaker mixtures with synchronized multi-view video. Each scene also contains preceding A-only and B-only stages that provide in-scene speaker references. It contains 77 scenes and 7,598 clips (11.84 hours), including 6,042 dual-annotated mixtures. After processing there leave 6,038 evaluable mixtures and 12,076 target-speaker rows. We additionally curate VOXBLINK2-AVSE from VoxBlink2, comprising 250,828 synchronized audio--lip-ROI pairs from 28,421 identities and 766.17 hours of speech. Our extractor uses frozen, 1,280-dimensional projected AV-HuBERT features, target-conditioned training, and layer-wise feature modulation. We jointly evaluate content with Qwen3-ASR-1.7B CER and target identity with WeSpeaker ResNet34 plus Overlapped Speech Detection (OSD). On the complete manifest, the best archived checkpoint obtains 0.2261 CER, 82.22% strict output correctness, and 69.53% both-output strict success.
Audio-visual speech enhancement (AVSE) uses a target speaker's visible articulation to recover that speaker's voice from overlapped speech. Yet most evaluation protocols rely on synthetic mixtures and reliable video. The ISCSLP 2026 Real-World AVSE Challenge addresses both gaps. Track~1 combines naturally recorded two-...
Kai Li, Wen-Ze Ren, Jun-Jie Li et al.· 0 citations
Online audio-visual target-speaker extraction aims to remove competing voices while preserving speech quality and bounding lookahead. Existing extractors are built and evaluated for separation on synthetic mixtures, leaving listening quality and meeting behavior largely untested. We introduce Audio-Visual Personalized...
Rayhan Rashed, Senja Filipi, Ross Cutler· 0 citations
This work proposes an LLM-based target-speaker voice activity projection (TS-VAP) module, which forecasts the future activity of the target and conversational partner directly from the overlapping mixture, drawing on the linguistic and conversational knowledge of a speech-LLM, and uses this prediction to guide a low-la...
A review of the current landscape and open problems in multimodal TSE from the perspective of auxiliary target cues drawn from multiple modalities and summarizes future directions concerning adaptive cue fusion, instruction-driven extraction, realistic evaluation, and trustworthy deployment.
Xin-Yuan Qian, Yang Zhou, Zi-Yang Jiang et al.· arXiv.org· 0 citations
B BabelFake is introduced, a multilingual audio-visual DeepFake benchmark recorded with consenting participants, and it is shown that detection difficulty depends on the audio-visual generation pairing, with substantial performance degradation when authentic audio is preserved.
Carlotta Segna, Joel Tschesche, Anna Rohrbach· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.