Analyzing structural and semantic similarities between formal business process models using ChatGPT-5.1: a test report

Abstract

Large Language Models (LLMs) are emerging as a promising tool in Business Process Management for comparing and validating process models. In this study, we evaluated an LLM’s ability to compare reference models with systematically modified variants representing typical modeling mistakes as well as harmless variations, such as layout or wording changes. The results show that the applied LLM can reliably detect structural and semantic differences between formal business process models using Business Process Model and Notation, while distinguishing them from acceptable variations, demonstrating strong potential for automated model validation. However, the LLM’s performance and accuracy are influenced by factors such as model complexity, the number of inserted modifications, and the total number of models and modifications provided simultaneously. High reliability is achieved when models are presented in a standardized, semi-structured format and supported by clear prompting instructions. Even multiple models can be processed effectively, up to a certain threshold of total modifications. Overall, the findings suggest that generative Artificial Intelligence tools for natural language processing, such as LLMs, may provide meaningful support in process model validation, offering efficiency gains and a level of abstraction that exceeds manual comparison.