LLM-Based Instance Model Generation via Code Synthesis

Abstract

The automated generation of instance models plays a central role in object-oriented software testing, benchmarking of graph databases, and the assurance of cyber-physical systems. Model generators are tools designed to address this problem, typically taking as input a set of constraints that the generated models must satisfy. Traditional model generators rely on SAT solvers, search-based techniques, grammar-based approaches or learning-based generative techniques. In this paper, we investigate the potential of Large Language Models (LLMs) as a new paradigm for instance model generation. We propose an approach that reformulates the generation problem as a code generation task, on which the LLMs excel. Specifically, meta-models are encoded as Pydantic models, while well-formedness constraints are expressed using data validation Pydantic constraints. The LLM is prompted to generate executable code that constructs valid instance models, and a feedback loop is employed to iteratively correct invalid outputs. We evaluate our approach along four dimensions—scalability, consistency, diversity, and realism—across two use cases and three LLMs, and compare it against Refinery, a state-of-the-art model generator. The LLM-based approach demonstrates strong scalability with respect to instance model size, being able to generate models exceeding 2000 elements by effectively leveraging programmatic constructs such as loops. In terms of consistency, one of the evaluated LLMs achieves a high probability of generating fully consistent models, even for very large instances. However, diversity emerges as a major limitation of LLM-based generation, with the proposed generator showing a significant drop in diversity as the scope increases. Finally, while the LLM-based generator exhibits a certain degree of realism, its performance in this dimension is influenced by the domain of the meta-model.