Skip to content

Do Audio LLMs HEAR Pain? Tracing Acoustic Evidence from Speech to Language Reasoning

Oct 2026 · 0 citations · 29 references
Computer Science

TL;DR

HEAR is introduced, a four-stage diagnostic framework that traces pain-related information from the speech signal to the model's prediction, and applies it to seven closed-source and open-weight LALMs, indicating that current LALMs encode paralinguistic information in early layers that is largely unavailable when the model produces its answer.

Abstract

Large audio language models (LALMs) can process speech directly, but it remains unclear how well they use the non-lexical acoustic information that speech carries. We study this question through pain, using TAME-Pain, which contains pain-neutral Harvard sentences and spoken pain ratings recorded at different self-reported pain levels. We introduce HEAR, a four-stage diagnostic framework that traces pain-related information from the speech signal to the model's prediction, and apply it to seven closed-source and open-weight LALMs. On the Harvard sentences, where pain can only be inferred from the voice, all models perform at chance level, and same-speaker references or few-shot examples improve balanced accuracy by no more than six points. On the spoken ratings, the same models reach 87.5-99.7% balanced accuracy. Layer-wise probing of open-weight LALMs shows that pain can be decoded from early audio representations (74.1% balanced accuracy for binary prediction and 45.0% for three-class severity), but that this information, along with most pain-associated acoustic cues, decreases in deeper layers. In an acoustic perception test, weaker models cannot distinguish even large changes in these cues, whereas stronger models can but still fail to infer pain. Finally, we recover pain-related acoustic cues and directly reinject them as structured text into the prompt, which improves pairwise pain prediction by up to 32 points. These results indicate that current LALMs encode paralinguistic information in early layers that is largely unavailable when the model produces its answer.

View source

Similar papers

Preprint Oct 2026

One Waveform, Two Stories: Benchmarking Speech-Acoustic Conflict in Audio-Language Models

Audio-language models (ALMs) are evaluated on their understanding of both speech and environmental sounds. What remains underexplored is how they respond when spoken claims contradict acoustic evidence within the same recording. We introduce SAC-Bench, a benchmark of intra-audio speech--acoustic conflict comprising 616...

Wen-Bin Wang, Yu-Heng Li, Koichiro Yoshino · 0 citations
Open access Nov 2026

Audio large language models align with human neural responses during spoken-language comprehension.

Audio large language models (audio LLMs) process continuous speech directly and transform it into progressively more abstract representations, providing a useful computational point of comparison with human spoken-language comprehension. Previous studies have reported model-brain correspondence for pretrained speech mo...

Shi-Yao Zhu, Ao Chen, Yan-Lu Xie · 0 citations
#natural language process... Preprint Oct 2026

VocalGrad: Evaluating Acoustic Perception in Audio Language Models

Recent large audio language models (LALMs) have extended their capabilities beyond lexical speech understanding to tasks requiring paralinguistic and non-linguistic audio understanding, such as emotion recognition. However, these models still struggle to reliably utilize acoustic information, particularly when it co-oc...

Koki Ryu, Hitomi Yanaka · 0 citations
#artificial intelligence Preprint Oct 2026

Relevant Evidence Decoding for Audio-Visual Hallucination Mitigation

Audio-Visual Large Language Models (AV-LLMs) remain prone to cross-modal hallucinations, where one modality incorrectly affects predictions about another. Although contrastive decoding reduces hallucinations in vision-language models, its direct extension to AV-LLMs overlooks a key challenge: different questions requir...

Hyunjae Ra, Aecheon Jung, Jungin Park et al. · 0 citations
#natural language process... Preprint Nov 2026

ReListen: Training-Free Evidence Localization in Hour-Long Audio with Frozen Audio Language Models

ReListen is a training-free method in which a frozen audio language model finds its own evidence before answering, and its memory stays constant even for recordings too long to hear whole on one GPU, therefore, a frozen audio language model can search hour-long recordings by itself, without a transcript or additional t...

Ming-Hao Hsu, Shu-Jie Liu, Yuxuan Hu et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.