A Transformation-Based Benchmark for Evaluating the Robustness of LLMs in Generating OCL

Abstract

Large Language Models (LLMs) have shown promising performance in generating Object Constraint Language (OCL) constraints from natural language specifications. However, existing evaluations rely on publicly available UML models, which may overestimate generalization due to potential data leakage and reliance on recurring lexical and structural patterns. This paper introduces a transformation-based benchmarking approach for OCL generation, based on deterministic, semantics-preserving UML model transformations; such as identifier renaming, attribute reification, and association reification, that preserve specification intent while altering model representation. We evaluate a diverse set of closed-source and open-source LLMs under zero-shot, few-shot, and chain-of-thought prompting, measuring syntactic accuracy (well-formed and type-correct OCL) and semantic accuracy (correct interpretation with respect to the UML model). While leading models achieve high performance on non-transformed models (up to 92.10% syntactic accuracy and 74.56% semantic accuracy), performance degrades substantially under transformation, with best semantic accuracy dropping to 55.96% for closed-source models and 42.20% for open-source models. These results demonstrate that current LLMs remain brittle under semantically equivalent but structurally altered models, suggesting a sensitivity to surface-level patterns and highlighting the need for transformation-based benchmarks for evaluating OCL generation.