Assessing Capabilities of Large Language Models for Multi-Level Modeling

Abstract

Large Language Models (LLMs) are increasingly being explored in Model-Driven Engineering as assistants for generating, transforming, and evolving modeling artifacts from natural language descriptions. Existing work, however, has mainly focused on traditional two-level modeling settings, while the applicability of LLMs to multilevel modeling remains largely unexplored. Multilevel modeling introduces specific challenges, including reasoning across multiple abstraction levels, handling deep instantiation, preserving cross-level consistency, and distinguishing between types, instances, and linguistic classifications. This paper investigates the capabilities and limitations of LLMs’ chatbots in supporting multilevel modeling tasks from natural language descriptions. We propose a scenario-based evaluation framework that combines two representative case studies from the MULTI challenge series with a conventional two-level modeling benchmark. The framework evaluates LLMs across six representative MLM tasks, covering hierarchy navigation, additive and subtractive model evolution, structural modifications, instance generation, and complete hierarchy synthesis. The results show that LLMs can support exploratory MLM tasks, but their outputs remain candidate artifacts that require expert validation, especially when correctness depends on cross-level reasoning and the preservation of consistency.