Skip to content
Open access

Audio large language models align with human neural responses during spoken-language comprehension.

Nov 2026 · NeuroImage · pp. 122343 · 0 citations · 46 references
Medicine

Abstract

Audio large language models (audio LLMs) process continuous speech directly and transform it into progressively more abstract representations, providing a useful computational point of comparison with human spoken-language comprehension. Previous studies have reported model-brain correspondence for pretrained speech models and text-based language models, but it remains unclear which internal representations of audio LLMs are most closely associated with neural responses during online listening. We compared EEG responses from 41 native Mandarin listeners with internal representations from Qwen2-Audio, Qwen2.5-Omni, and Kimi-Audio in a Mandarin auditory semantic-violation paradigm that controlled sentence structure and event boundaries. We extracted speech-related and word-level semantic-update representations and quantified their incremental contributions to held-out EEG prediction beyond lower-level controls. Audio LLM representations improved EEG prediction beyond the acoustic envelope, phoneme onsets, and word onsets (Δr = 0.012, corrected P = 0.028). Prediction-score partitioning indicated that semantic violation primarily increased the overlap in EEG-predictive information between speech-related and semantic-update representations, rather than the unique contribution of either representation class. This pattern was relatively stable in Qwen2-Audio and Qwen2.5-Omni and directionally consistent, although statistically weaker, in Kimi-Audio. These findings provide component-level evidence that audio-conditioned model representations correspond to human neural responses during spoken-language comprehension, without implying mechanistic equivalence.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.