The proposed ToolEX (Tool Equivalent eXpansion), a framework that automatically discovers and annotates the tool combinations functionally equivalent to the labeled ones, and confirms that the one-to-one problem extends beyond tool retrieval.
Abstract
In open-world scenarios with massive and evolving tool repositories, tool-augmented large language models rely on a retriever to surface relevant tools for a given query. Because such repositories often contain many tools that implement the same functionality, a single query can often be resolved by several distinct but functionally equivalent tool combinations, making the natural query-to-tool mapping inherently one-to-many. However, existing tool retrieval benchmarks annotate each query with a single relevant tool combination, collapsing this one-to-many mapping into a rigid one-to-one annotation and causing valid retrieved tools to be misjudged as failures. To address this, we propose ToolEX (Tool Equivalent eXpansion), a framework that automatically discovers and annotates the tool combinations functionally equivalent to the labeled ones. Applied to the 7,360-query Tool-DE benchmark, ToolEX finds that 67.9% of sub-queries admit equivalent alternatives, expanding the singular ground truth to an average of 5.3 valid combinations per query. Using the expanded benchmark ToolEQ, we re-evaluate eight base retrievers and two fine-tuned variants; metrics on ToolEQ rise substantially over Tool-DE, showing that one-to-one annotation systematically underestimates retrievers and that 30--47% of the reported fine-tuning gain is an evaluation artifact rather than genuine improvement. Applying the same pipeline to skill retrieval on SkillRet further confirms that the one-to-one problem extends beyond tool retrieval.
Tool-augmented LLM agents discover relevant APIs from large registries before planning execution, yet finite context windows prohibit injecting all tool definitions into a single prompt. The dominant approach, which we term Retrieve-then-Plan (RtP), embeds the user query and retrieves similar tools via approximate near...
Ming-Fei Zhang, Amit Surana, Ziauddin Majid et al.· 0 citations
A novel chatbot architecture leveraging OpenAI's GPT-4 model for automated extraction and analysis of repository data is introduced, suggesting that this architecture can make repository data more accessible to technical and non-technical audiences through the production of actionable insights.
Muhammad Jawad Chowdhury, Md. Sakib Khan· 1 citation
Processing technical terms in documents is essential for digital libraries and knowledge organisation, supporting applications such as scholarly search, information retrieval, and domain terminology construction. While document-level term identification, a sub-task of automatic term extraction (ATE), has been tackled a...
Rei Miyata· ACM/IEEE Joint Conference on...· 0 citations
This work asks whether making a related but unnecessary tool available affects a model's ability to answer from its own knowledge, and whether a preceding tool interaction changes this behaviour.
Saanvi Paturi, Arsen Kenzhebayev, Arham Sethi et al.· 0 citations
This work generates natural-language descriptions for opaque columns with a language model at ingestion, from their names, statistics, and sample values, and stores them as curator-reviewable catalog metadata for search and a natural-language visual-discovery interface.
Caio N. Maia, A. N. Maia, Pedro L. P. Corrêa· ACM/IEEE Joint Conference on...· 0 citations
It is indicated that retrieval-stage filtering can reduce the candidate action space exposed to agents before execution, complementing downstream tool-use safeguards, and providing a conservative operating point for safety-critical deployments.
Qinfei Li, Xiaoxuan Dong, Jin Zhang et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.