A tool selection mechanism for LLM-based model management agents

Abstract

In Model-Based Engineering (MBE), practitioners frequently have to choose appropriate tools from many different options. LLM-based agents are software components that depend on Large Language Models (LLMs) to autonomously select and apply software tools to perform specific tasks. Although LLMs have already been used in the MBE context, LLM-based agents to assist users of MBE tools remain underexplored. This is particularly true in industrial environments where only medium-sized on-premise LLMs can be considered due to policies related to security or data privacy. To investigate the potential of LLM-based agents for MBE, we study model management tools for transformation and editing. Currently, off-the-shelf agents such as Microsoft Copilot can invoke model management tools when the task is explicitly described. However, these agents struggle to select the correct transformation or operation when they only have limited contextual information, especially when coupled with medium-sized LLMs. To overcome this, we propose an approach based on complementary mechanisms. First, we provide a server and associated LLM-based agent with dedicated tools for each transformation available on this server. We also provide a similar server and agent for model editing operations. Then, to enable the two agents to efficiently select transformations and operations (respectively), we rely on a tool retrieval technique based on a tool relevance score computed by an LLM. We evaluate these agents using generated model management datasets that we contribute to the community. The obtained results show that our LLM-based agents select more accurately modeling tools to perform user instructions than off-the-shelf agents.