A Comparative Study of LLMs and Human Judgment in UML Diagram Evaluation

Abstract

: Large Language Models (LLMs) are increasingly used to evaluate software engineering artifacts. This paper investigates the reliability of LLMs in evaluating UML diagrams generated through reverse engineering processes (source code). We ask: do LLM assessments align with those of human experts? A COMAS-based framework is proposed for trust-aware LLM selection. In the original COMAS, indirect trust is computed via Dijkstra max-product chains. We replace this with direct human–LLM comparisons for each criterion independently. This is valid since LLMs can evaluate any diagram on demand. Trust is measured using an L1-norm-based divergence that is robust under small-sample conditions. The developed framework supports flexible per-criterion LLM selection via expert-defined thresholds and weights. An empirical study involved 30 students evaluating five UML sequence diagrams. Four evaluation criteria were used: completeness of requirements description, non-contradiction to requirement specification, UML notation compliance, and Miller’s principle. Four LLMs were compared: Groq, Gemini, Mistral, and Copilot. For 90% of participants, a combination of two or three LLMs outperformed any single model. Hybrid per-criterion LLM selection is a viable strategy for automated UML diagram evaluation.