Organisations keep decades of financial records as scanned PDFs on shared drives, and AI agents can now reach those drives through Model Context Protocol (MCP) servers. Existing benchmarks do not mirror this production setting because they simplify the task. In some, the relevant documents are handed to the agent, so i...
Ritiz Tambi, S. Shayegan, Joshua Meyer et al.· 0 citations
VAmoS Energy, a benchmark that combines challenges in 100 calls about utility billing and payment assistance, is introduced, showing why voice agents need evaluation across the whole call, including what they say, what they change, and how they handle competing speech.
Joshua Meyer, S. Shayegan, Ritiz Tambi et al.· 0 citations
VAmoS Bench, the Voice Agent Simulation Bench, measures complete voice-agent systems end to end in a stateful customer-support task and catches an agent that claims to have changed a card without updating the database as well as one that makes the right database change while disclosing protected information.
Joshua Meyer, S. Shayegan, Ritiz Tambi et al.· arXiv.org· 5 citations· ⚡2
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
