Human face-to-face conversation requires coordinating speech, gestures and expressions across both speaking and listening. We present ELLSA 2, an end-to-end full-duplex dialogue model that jointly optimizes speech interaction and streaming gesture-expression generation. An autoregressive, chunk-wise synthesizer combine...
Si-Yin Wang, Wen-Yi Yu, T. Chiba et al.· 0 citations
SALMONN-duo is proposed, an adaptive dual-system voice agent inspired by dual-process theories of cognition that separates real-time interaction from deliberative computation by pairing an always-on, fast-thinking full-duplex speech LLM with a powerful asynchronous slow-thinking LLM agent.
Wen-Yi Yu, Si-Yin Wang, T. Chiba et al.· 1 citation
This Review provides a comprehensive review of recent progress in integrating audio into large language models, with a focus on four key areas: audio comprehension, audio generation, speech-based interaction and audio–visual understanding.