MultiCAT-Bench, the first benchmark focused on detailed categorization of tasks for assessing tool utilization, is introduced, along with an approach for its automated generation, which utilizes the GPT-5 family.
Abstract
Reliable function calling (a.k.a. tool use) is a core capability of LLM agents. However, existing evaluations insufficiently probe how robustness varies with task complexity. We introduce MultiCAT-Bench, the first benchmark focused on detailed categorization of tasks for assessing tool utilization, along with an approach for its automated generation, which utilizes the GPT-5 family. MultiCAT-Bench spans ten principled categories of difficulty with 3 789 test cases. Using this dataset, we evaluate ten LLMs from 9 model families. The analysis revealed that Recall metrics (overall ~72.4%, tool name identification ~81%, arguments ~90.4%) are lower than Precision (overall ~88.3%, tool name identification ~99.8%, arguments ~93%) across the examined categories. This indicates that models are less likely to extract relevant information, but when they do, they achieve higher accuracy. Regarding the selected categories, the greatest impact on models’ accuracy was exerted by the number of calls in the response (average drop by a factor of 1.59), parameter optionality status (by a factor of 1.37), and the number of parameters in the function (by a factor of 1.22). The Grok 4.1 Fast and GPT-5 Mini models achieve the best average accuracy, 83.8% and 83.7%, respectively, across the benchmark.
Canary tools are introduced: diagnostic probe tools planted in an agent's Model Context Protocol (MCP) tool set, each engineered to probe one specific tool-selection weakness, evidence that the probes measure reasoning, not phrase-spotting.
A multi-agent testing framework in which requirement analysis, test-case generation, sandboxed execution, and defect detection are delegated to four distinct agents, and in which detection rests on a composite rule combining the execution signal with the semantic judgment of a dedicated diagnostic agent.
Yuxuan Li· Transactions on Computing Sc...· 0 citations
A paired benchmark that tests whether LLM agents exhibit conscious allocation behavior under a fixed budget in two contexts: an abstract text-based formulation and a code-construction task finds that every frontier model testedacts near-optimally in the abstract framing but fails to transfer this ability to script-writing.
As LLMs increasingly act through tools, they must reconcile user instructions, parametric knowledge, and dynamic environmental observations before taking actions. We introduce KC-Bench, a controlled multi-turn benchmark for measuring this capability across world-knowledge conflicts, input inconsistencies, and multi-source temporal conflicts. Its 238 tasks are manually screened from more than 1,000 generated candidates and combine a user simulator, stateful tools, deterministic environment assertions, an open-source natural-language evaluator, and human trajectory verification. Evaluation of nine models, including DeepSeek-V4-Flash, GLM-5.2, and MiniMax-M3, shows substantial cross-domain variation: no model handles factual correction, identity consistency checking, and temporal conflict resolution reliably across all settings. In the simulated environments, missed conflicts can propagate to tool calls or synthetic protected-data flows. KC-Bench isolates this model-level behavior rather than ranking complete agent frameworks, and provides a reproducible diagnostic for developing conflict-aware reasoning and execution safeguards.
Yaxing Lyu, Sheng-Jie Zhou, B. Toh et al.· 0 citations
The first benchmark to systematically study LLM-as-a-judge reliability for agentic tool-calling over workflow DAGs, as distinct from the broader LLM-as-a-judge task of open-ended text or preference evaluation, exposes fundamental limitations of current LLM judges and yields practical guidelines for reliable evaluation in agentic systems.
Abhigya Verma, Amit Kumar Saha, Seganrasan Subramanian et al.· 0 citations
CompressAgent is introduced, an environment-verified benchmark for ACC compression across nine independently constructed ACCs, three task families, three fixed Qwen API model identifiers, six retained-context budgets, and 15,525 runs, uncovering a nonlinear, method-dependent reliability frontier.
Ying-Han Hou, Zong-You Yang· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
