Review

Towards general auditory intelligence for machine listening and speaking

Si-Yin Wang Zengrui Jin Changli Tang Qiujia Li Bo Li Chen Chen Yu-Chen Hu Wen-Yi Yu Yixuan Li Jimin Zhuang Yudong Yang Ming-Qiu Wang Michael Han Yifan Ding Ju Bai Tom Y. Ouyang Shuo-yiin Chang Xian-Zhao Chen Xiao-Hai Tian Jun Zhang Lu Lu Guang-Zhi Sun Zhe-Huai Chen Ji Wu Bo-Wen Zhou Yuxuan Wang T. Sainath Yong-Hui Wu C. Zhang

Aug 2026 · Nature Machine Intelligence · Vol 8, pp. 1201 - 1220 · 1 citation · 273 references

TL;DR

This Review provides a comprehensive review of recent progress in integrating audio into large language models, with a focus on four key areas: audio comprehension, audio generation, speech-based interaction and audio–visual understanding.

DOI Semantic Scholar

View source

Similar papers

Preprint Aug 2026

HUG-VIS: A Multimodal Benchmark for Human-centered Understanding and Generation in Visual Intelligence

HUG-VIS, a unified benchmark for Human-centered Understanding and Generation in Visual Intelligence, contains 8,400 seated half-body videos of 30 professional actors, each performing the same 280 emotion-action-prompt assignments under a controlled Mandarin studio protocol, with synchronized video, audio, text, and alp...

Fei Ma, Ze-Bang Cheng, Ming-Hui Li et al. · 1 citation

#artificial intelligence Preprint Sep 2026

Listen Then Reason: Perception-Grounded Test-Time Reinforcement Learning for Large Audio-Language Models

Perception-Grounded TTRL (PG-TTRL) is proposed, which aligns label-free test-time optimisation with perceptually grounded reasoning, encouraging the model to structure its reasoning more strongly on the audio input, showing the value of perceptual-grounding optimisation for test-time audio reasoning.

Jia-Heng Dong, Xiao-Feng Yu, Jean Honorio et al. · 0 citations

#machine learning Preprint Aug 2026

MRMAD: A Multi-Round Multi-Audio Benchmark for Evaluating Acoustic Degradation Perception in Large Audio-Language Models

This work introduces MRMAD, a Multi-Round Multi-Audio Degradation benchmark for evaluating audio degradation perception and understanding in LALMs, and finds that current models often recognize coarse content while failing to diagnose, compare, or reason about degradations reliably.

Yi-Ze Li, Ning-Yuan Yang, Sile Yin et al. · 2 citations

#artificial intelligence Preprint Aug 2026

EXAM2: Extending Audio Understanding in Multilingual and Multimodal Analysis

This paper introduces EXAM, a benchmark for multilingual and multimodal audio understanding spanning six languages and multiple modalities, and evaluates state-of-the-art open-source and proprietary LALMs as well as multimodal LLMs, revealing substantial performance gaps in multilingual and cross-modal understanding.

Jia-Wen Wang, Xiao-Xue Gao, Ziliang Pang et al. · 0 citations

Book Open access Oct 2026

Auditory Feedback as a Communication Modality: Modeling Perceived Semantic Expression of UX Sounds

This work identifies recurring audio design patterns and investigates how they relate to perceived semantic expression across the communication levels of system status, affect, and brand identity, then applies explainable machine learning to a predictive model capturing interpretable parts of this relationship.

Annika Frommholz, S. Lepa, S. Weinzierl et al. · Proceedings of the 28th Inte... · 0 citations

#artificial intelligence Preprint Oct 2026

Relevant Evidence Decoding for Audio-Visual Hallucination Mitigation

Audio-Visual Large Language Models (AV-LLMs) remain prone to cross-modal hallucinations, where one modality incorrectly affects predictions about another. Although contrastive decoding reduces hallucinations in vision-language models, its direct extension to AV-LLMs overlooks a key challenge: different questions requir...

Hyunjae Ra, Aecheon Jung, Jungin Park et al. · 0 citations