Voice agents depend on accurate automatic speech recognition (ASR) to act on what callers say, yet ASR is evaluated on read, English-centric speech with word error rate (WER), which penalizes surface rather than semantic differences. We introduce mu-bench, a dataset of 4,270 caller utterances from 250 phone calls to an AI banking agent in English, Spanish, Turkish, Vietnamese, and Mandarin, centered on form-field inputs such as names, email addresses, and confirmation codes. We release Utterance Error Rate (UER), an LLM judge of whether a transcript preserves meaning, calibrated against human raters, together with an LLM normalizer that makes WER comparable across providers'output formats. On 1,847 human-rated transcripts, UER agrees with annotators at $\kappa$ = 0.78, versus 0.53 for exact-match WER on normalized text. We rank six commercial providers on a public leaderboard; the best reaches 11.9% UER, and Mandarin is hardest for all six.
The resulting 82M-parameter model, Wayu-Paxa-TTS-Edge, enables on-device Thai TTS without reference audio and achieves the lowest pause-placement error and intra-word pause rates among the three systems.
Kunat Pipatanakul, Potsawee Manakul, Warit Sirichotedumrong et al.· 1 citation
This work proposes Agentic-GER, an LLM-based agent for terminology correction in long-form speech, which uses global context from the full transcript to identify suspicious terms and resolve ambiguous hypotheses.
Yan-Qiao Zhu, Wu-Peng Wang, Zhifu Gao et al.· 0 citations
This work curates a human-annotated topic-utterance judgments dataset and examines the superior performance of lightweight LLM matchers over embedding and regex models when equipped with natural language descriptions.
Saman Rahbar, Xi-Liang Zhu, Irvin Cardoza et al.· 0 citations
FA-Bench is presented, an open framework that fixes choices once and releases the code, splits, phone mapping, text normalization and scoring script, with results published periodically, and discovered systematic bias in how current systems time words.
Wei Chu, Yuan-Zhe Dong, Ke Tan et al.· 0 citations
This work presents AVERT, which scores each candidate value by combining cross-turn agreement with a trained audio-conditioned verifier and resolves the three error types with three operators, vote, add, and swap, each restricted to the slots where its error is common.
Sharing sensitive speech corpora demands deidentification beyond text redaction: the voice itself is a biometric identifier, so redacting a transcript does not anonymize the recording. We evaluate a two-layer pipeline on an 18.7-hour Indonesian interview corpus on gender bias spanning eleven ethnicities. Layer 1 redact...
M. T. Uliniansyah, Siska Pebiana, Andi Djalal Latief et al.· International Conference on...· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduSep 24, 2026
A new method, called CW-Net, translates the reasoning process of an autonomous vehicle’s AI system into understandable concepts that explain its behavior.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
