A conformance-checker-driven framework for reliable natural-language-to-SysMLv2 translation that enforces production-level acceptance as the termination condition and converts probabilistic outputs into production-accepted SysMLv2 artifacts suitable for loading, visualization, and engineering use.
Abstract
Model-Based Systems Engineering (MBSE) relies on formal system models as primary technical artifacts for representing requirements, structure, and behavior across the system lifecycle. With the standardization of SysMLv2 as a textual language, interest is increasing in translating natural-language descriptions directly into executable models. For practical deployment, generated models must be accepted by industrial modeling environments, not merely satisfy grammar constraints. We present a conformance-checker-driven framework for reliable natural-language-to-SysMLv2 translation that enforces production-level acceptance as the termination condition. The system embeds a SysMLv2 conformance checker within a generate-check-repair loop. Each model is evaluated using the checker, and deterministic diagnostics are incorporated into revisions until zero conformance errors are achieved. Using the production checker as the oracle ensures the framework targets deployability rather than grammar plausibility. We evaluate the approach on the full SysMBench prompt set of 151 prompts across four large language model backends, yielding 604 prompt-model cases. Single-shot generation achieves 51.16% production-conformance acceptance, while our approach achieves 100.00% conformance. By elevating production conformance from a post-processing check to a control mechanism within generation, the framework converts probabilistic outputs into production-accepted SysMLv2 artifacts suitable for loading, visualization, and engineering use.
This work proposes TraceDev, a multi-agent framework for automated software development grounded in use cases that contain multiple functional points and complex semantics, and demonstrates the effectiveness of TraceDev in repository-level code generation from requirements.
The results indicate that current general-purpose LLMs can achieve practically significant performance on the unstructured NL-to-LTL task without task-specific fine-tuning, and suggest that modern LLMs are becoming viable front-end assistants for semi-automated formalization workflows.
: Model-Driven Software Engineering has long excelled at generating code from static structural models, yet the specification and generation of dynamic behavioral models remains a persistent challenge. Meanwhile, Large Language Models (LLMs) offer flexible, natural-language based code generation but suffer from non-determinism and hallucinations. This paper presents a semi-formal approach that bridges these two paradigms for behavioral modeling via UML state machines. We contribute a textual modeling language that captures the essential elements of UML state diagrams—states, transitions, events, guards, and entry/exit actions—alongside a deterministic code generator that transforms state machine models into Java code following the Gang of Four State design pattern. The language supports two complementary action annotation styles: direct code fragments for concise, self-contained actions, and natural language descriptions for semantically richer behavior to be completed by an LLM weaver. LLM involvement is deliberately scoped to small, well-constrained action bodies, reducing token consumption and non-determinism compared to fully LLM-based approaches. Validated through the Gumball Machine case study, correctness is confirmed by automated tests covering state and transition coverage criteria, and repeating the LLM weaving step produced consistent results across all runs. Compared to both classical UML tooling and fully LLM-based generation, the approach offers stronger determinism, better traceability, lower cognitive modeling effort, and reduced computational cost, while retaining the flexibility to express complex action behavior in natural language where formal specification would be unnecessarily burdensome.
Oliver Engling, Felix Schwägerl, Thomas Buchmann· International Conference on...· 0 citations
The need for custom validation capabilities to ensure that models produced by both humans and LLMs are trustworthy enough to serve as the foundation of the MBSE ecosystem is addressed.
Á. Tóth, Levente Bajczi, Ármin Zavada et al.· Proceedings of the ACM/IEEE...· 0 citations
This paper addresses automated unit test generation with large language models (LLMs). LLM-based test generation has not yet attained a quality level sufficient for practical use in industry. Although LLMs often reproduce API syntax faithfully, they frequently disregard semantic usage constraints and execution-environment dependencies, leading to assertion failures, mock-related errors, and reference/resolution errors. A prior failure analysis of Java unit test generation using GPT-4o classified 2980 trials into eight failure patterns and identified three root-cause mechanisms: external context ignorance, internal context ignorance, and a syntax–semantics gap. Building on that analysis, this paper proposes a prompt design comprising three strategies: (1) making the execution state explicit in the generated test, (2) stating semantic constraints explicitly, and (3) injecting environment constraints prior to generation. In contrast to generic techniques such as few-shot learning or chain-of-thought prompting, each proposed strategy is tied to a specific root-cause mechanism, yielding a systematic design in which each rule is explicitly justified by its correspondence to a specific root-cause mechanism. Experiments on 298 methods with five models (GPT-4o, GPT-5, GPT-5.1-Codex, Claude Sonnet 4.5, and Gemini 2.5 Pro) show improved test execution success rates for every model, with absolute gains ranging from 1.1 to 21.1 percentage points (pp). Mock-related errors were reduced by 61.9%–99.2% relative to the baseline prompt, demonstrating effectiveness against the targeted failure patterns. Finally, conditions under which the strategies transfer to other code-generation tasks are discussed, along with limitations on their scope.
