Skip to content
Preprint

Beyond Accuracy: ARIA-Rubrics for Evaluating Audio Reasoning in Large Audio Language Models

Sep 2026 · 0 citations · 35 references
Computer Science

TL;DR

This work proposes ARIA-Rubrics (Audio Reasoning Integrity Assessment), a lightweight, annotation-free gold reasoning chains, automatic and transparent framework comprising six complementary metrics that evaluate audio reasoning quality across perceptual grounding, reasoning coherence, and answer consistency.

Abstract

Large Audio Language Models (LALMs) have shown strong performance on audio reasoning benchmarks, but accuracy alone cannot distinguish true reasoning from superficial pattern matching, often overestimating reasoning ability since high scores may result from guessing rather than genuine audio understanding. Evaluating the reasoning process itself is essential for improving LALMs'reasoning ability, yet remains challenging. Existing methods either rely on costly human annotation or opaque LLM-as-judge approaches, making them impractical, biased, and lacking transparency. Moreover, audio reasoning introduces unique challenges absent in text-based settings, perceptual hallucination and cross-modal alignment between audio understanding and textual inference, hence text-based evaluation frameworks cannot be directly applied. Therefore, we propose ARIA-Rubrics (Audio Reasoning Integrity Assessment), a lightweight, annotation-free gold reasoning chains, automatic and transparent framework comprising six complementary metrics that evaluate audio reasoning quality across perceptual grounding, reasoning coherence, and answer consistency. We use Chain-of-Thought prompting as an externalization mechanism to make the reasoning process observable. Experiments on 9 models across 2 benchmarks identify three reasoning modes of current LALMs with actionable directions for future development, with ARIA-Rubrics achieving high correlation with human judgments. The code is available at the Github Repository.

View source

Similar papers

MISHAP-Bench: A Hallucination Benchmark for Large Audio-Language Models

This work introduces MISHAP-Bench, a comprehensive benchmark with 12,000 challenging open-ended question-audio pairs and a rigorous evaluation pipeline covering two hallucination categories, and proposes a groundedness judge that uses reference rubrics and judge prompts guided by human annotations.

Wen-Soi Zhi, Giulio Segalini, Jian-Jia Chen et al. · 0 citations
Open access Sep 2026

Probing Large Audio-Language Models for Compositional Understanding of Sounding Actions

Large audio-language models (LALMs) excel at understanding and reasoning tasks over atomic sound events, yet their ability to infer higher-level human activities from such fine-grained events remains largely unexamined. Everyday human actions and activities, such as setting a table, cleaning the house, or preparing a b...

Michel Olvera, Paraskevas Stamatiadis, Chang-Hong Wang et al. · 0 citations
#artificial intelligence Preprint Sep 2026

SEABED: SouthEast Asian Benchmark for Evaluating Audio Reasoning

This work introduces SEABED, an audio-first question answering dataset designed to benchmark language and audio reasoning models on SEA speech, and evaluates not only answer accuracy, but also whether models'stated reasoning is grounded in the audio evidence.

Harshit Rajgarhia, Asif Shaik, R. Lokesh et al. · 0 citations
#natural language process... Preprint Aug 2026

OCR-MetaReasoning Benchmark: Evaluating the Meta-Reasoning Ability of MLLMs in Text-Rich Image Understanding

Experiments with representative closed-source and open-source MLLMs show that OCR-grounded meta-reasoning remains far from saturated: models struggle with visible-rule application and layout-sensitive inference, while process-compliant rationales can accompany incorrect final answers under exact-match evaluation.

Geng-Xu Li, Yuan Wu, Yi Chang · 0 citations
#machine learning Preprint Sep 2026

Do Reasoning Representations Help Humans Evaluate LLM Outputs?

This work conducts a controlled human study of six reasoning formats across tasks of varying complexity, supported by a web-based framework that randomizes task domains, problem instances, and representation order.

Jaewoo Lim, Sungbok Shin, San Hong · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.