This study explores how LLMs can be harnessed to automatically translate a neutral graph representation of fluid system models into executable code for two widely adopted simulation environments: the Python library WNTR and the Modelica Standard Library.
Abstract
Large language models (LLMs) have demonstrated a strong ability to generate syntactically correct code from natural-language specifications. In this study, we explore how LLMs can be harnessed to automatically translate a neutral graph representation of fluid system models into executable code for two widely adopted simulation environments: the Python library WNTR and the Modelica Standard Library. We conduct a systematic comparison of ten state-of-the-art LLMs and six prompting strategies that differ in the contextual information supplied (e.g., code or documentation). For each configuration we assess the generated code using a suite of software-quality metrics and we validate the functional fidelity of the resulting simulation models by reproducing benchmark fluid system scenarios. Our findings offer concrete guidance for researchers and engineers seeking to integrate LLM-driven code synthesis into model-based design pipelines. While the best-performing configurations achieve acceptable syntactic quality, we observe substantial gaps remain in simulation fidelity.
HEPToolBench is introduced, a benchmark of 28 collider-simulation tasks scored by deterministic, task-specific scorers, plus a three-task structured-debugging extension, and moving syntax generation into deterministic software can substantially improve reliability for both small local and frontier models.
Two novel contributions are introduced: CodeEval and CodeQual, an open-source execution framework that provides researchers with a ready-to-use evaluation pipeline for evaluating and improving LLMs in software engineering contexts, encompassing both functional correctness assessment and subjective code quality evaluation.
An automated, multi-dimensional evaluation framework for C# code generation, applying it to four state-of-the-art LLMs: GPT, Gemini, Claude, and Grok is presented and a substantial gap between correctness and quality attributes is revealed.
Seyed Mohammad Mahdi Ghalandarian, Majid Bazargani, Masoumeh Taromirad· 0 citations
The automated generation of instance models plays a central role in object-oriented software testing, benchmarking of graph databases, and the assurance of cyber-physical systems. Model generators are tools designed to address this problem, typically taking as input a set of constraints that the generated models must satisfy. Traditional model generators rely on SAT solvers, search-based techniques, grammar-based approaches or learning-based generative techniques. In this paper, we investigate the potential of Large Language Models (LLMs) as a new paradigm for instance model generation. We propose an approach that reformulates the generation problem as a code generation task, on which the LLMs excel. Specifically, meta-models are encoded as Pydantic models, while well-formedness constraints are expressed using data validation Pydantic constraints. The LLM is prompted to generate executable code that constructs valid instance models, and a feedback loop is employed to iteratively correct invalid outputs. We evaluate our approach along four dimensions—scalability, consistency, diversity, and realism—across two use cases and three LLMs, and compare it against Refinery, a state-of-the-art model generator. The LLM-based approach demonstrates strong scalability with respect to instance model size, being able to generate models exceeding 2000 elements by effectively leveraging programmatic constructs such as loops. In terms of consistency, one of the evaluated LLMs achieves a high probability of generating fully consistent models, even for very large instances. However, diversity emerges as a major limitation of LLM-based generation, with the proposed generator showing a significant drop in diversity as the scope increases. Finally, while the LLM-based generator exhibits a certain degree of realism, its performance in this dimension is influenced by the domain of the meta-model.
Javier Polo-Gambín, José A. Ruipérez-Valiente, José Antonio Hernández López· Proceedings of the ACM/IEEE...· 0 citations
Findings indicate that cross-model collaboration offers a practical and parameter-efficient alternative to scaling up monolithic models for code generation and maintains competitive accuracy when only 20% of test cases are available for diagnostic feedback.
Jiangping Huang, Wen-Guang Ye, Weisong Sun et al.· ACM Transactions on Software...· 0 citations
LangChoiceBench is introduced, a project-level code-generation benchmark for measuring Python preference, recommendation-implementation consistency, and language diversity, and it is found that Python remains heavily over-selected, recommendation-implementation consistency is low, and smaller open-weight models generally show stronger Python preference and lower language diversity.
Lukas Twist, Twm Stone, Helen Yannakoudakis et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
