Large Language Models for UML Class Diagram Modeling: A Preliminary Empirical Evaluation

Large Language Models for Software Engineering Diagrams: A Systematic Review of UML and ER modelling

Large language models (LLMs) are increasingly applied to diagram-based software and data modelling. Among various modelling notations, UML and entity-relationship (ER) diagrams are the most widely adopted for software modelling and data modelling, respectively. Recent literature has investigated various applications of LLMs in diagram modelling; however, their effectiveness and limitations have not been extensively discussed. This systematic literature review analyses 64 studies published between 2023 and 2025, examining diagram coverage, modelling tasks, technical approaches, evaluation practices, and limitations. Our findings reveal significant concentration patterns and gaps. UML-based software modelling strongly dominates, with class diagrams receiving the most attention whilst behavioural diagrams and data modelling remain underrepresented. Diagram construction from natural language is the primary focus, with limited work on transformation, quality assurance, and consistency checking. GPT-based models are heavily prevalent, raising concerns about reproducibility and vendor dependence. Evaluation practices are heterogeneous, employing diverse metrics and custom datasets with limited benchmark reuse and inconsistent reporting of robustness and statistical significance. Common limitations include semantic inaccuracies, hallucinated diagram elements, sensitivity to prompt formulation, and reproducibility constraints. This survey provides the first systematic synthesis of LLM-based diagram modelling research, highlighting needs for standardised benchmarks, stronger evaluation protocols, broader diagram coverage, and techniques for improving semantic reliability and multi-view consistency.

Large Language Models for Requirements Engineering: A Cross-Task Empirical Evaluation

Requirements-related information is scattered across heterogeneous artefacts such as user feedback, developer discussions, and software repositories, making the extraction of actionable requirements knowledge labour-intensive and hard to scale. Large Language Models (LLMs) can support many Requirements Engineering (RE) activities, from classification and traceability identification to specification and explanation generation, but existing evidence is fragmented across tasks, artefact types, and evaluation settings, and studies rarely offer cross-task evaluations or replication packages. We present two complementary empirical studies evaluating LLMs across five RE-related activities. The first is a controlled experiment on five lightweight open-source LLMs for feedback-driven requirements classification and specification generation. The second is an exploratory industrial case study on two frontier LLMs for traceability link identification and traceability explanation generation using real project artefacts. Classification and traceability identification were assessed with quantitative metrics, and generation tasks through human evaluation. LLM performance is strongly task-dependent, ranging from moderate to high, and no single model consistently outperformed the others, indicating that effective adoption depends on selecting models and prompting strategies per task. Our contributions are: (i) the first cross-task empirical evaluation of LLMs spanning five RE-related activities, (ii) replication materials supporting reproducibility, and (iii) a broader understanding of the capabilities, limitations, and practical readiness of current LLMs for RE.

Engineering Practices in Open-Source Agent Frameworks: A Large-Scale Empirical Study

A large-scale empirical study of mainstream open-source agent frameworks from an engineering perspective, providing empirical evidence linking framework design choices to engineering risks and highlighting the need for stronger guidance and support in agent framework development.

Software comprehension in code-centric and model-driven settings: an experimental comparison of models and code

Developers spend a large portion of their time on software comprehension, making it essential to understand how different artifacts – models and code – support this activity. While prior research has primarily explored the use of models as complementary aids for understanding code, limited evidence exists regarding their standalone effectiveness in conveying software’s structural and behavioral aspects. This gap is particularly relevant in contexts such as Model-Driven Engineering (MDE) and Low-Code/No-Code (LCNC) approaches, where models often serve as the primary or sole representation of software. The paper reports on two complementary controlled experiments that investigate how models and code contribute to understanding software structure and behavior. The first experiment compares comprehension when only reverse-engineered UML models or source code were available (code-centric settings), mimicking scenarios in agile or AI-assisted development where models are outdated or absent. The second experiment focuses on model-driven settings, examining whether providing generated code alongside platform-independent models improves comprehension and whether differences in code generators influence performance. The results demonstrate that models and code achieve comparable overall correctness, and thus models alone may be sufficient in model-centric scenarios where access to code is limited or unavailable. The experiments also reveal a consistent structure-behavior comprehension gap, show differences in comprehension across tasks regardless of artifact type, and indicate that adding code to models can both clarify and confuse comprehension depending on context.

Toward generalizable unified modeling language automation: a dual case study on class and use case diagram generation

Manual generation of unified modeling language (UML) diagrams creates bottlenecks in agile development due to inconsistency and labor intensity. While large language models (LLMs) offer generative capabilities, existing solutions suffer from data scarcity and inadequate evaluation tools. To address this, we present a dual-LLM pipeline integrating lightweight specification generation with reasoning-oriented code synthesis. Uniquely, this framework employs a weighted multimodal validation module utilizing diverse vision-language models (VLMs) to assess diagrammatic fidelity. We further address the data shortage by releasing benchmark datasets comprising 5,000 class and 3,000 use case diagrams. Empirical results demonstrate a 95.8% rendering success rate for class diagrams and strong semantic alignment for use case models. By mitigating structural and behavioral reasoning conflicts, this research provides a replicable architecture and rigorous assessment methodology, establishing a robust foundation for scalable, artificial intelligence-driven automation in software engineering.

The Impact of Software Model Slicing on Software Model Completion with Large Language Models