Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Models
AV-STE is proposed, a modular streaming audio-visual front-end that restores corrupted semantic speech tokens from noisy audio and lip video before they reach the speech LLM, and remains entirely frozen, preserving its pretrained conversational capabilities.
Bella Godiva, Yeonju Kim, Man-Ro Yong
· 1 citation