Large Language Models for Software Models: An Empirical Evaluation in Game Software Engineering

Abstract

Replication Package This repository is the replication package for the paper "Large Language Models for Software Models: An Empirical Evaluation in Game Software Engineering" (under review). It contains all materials needed to reproduce the controlled empirical evaluation of seven state-of-the-art LLMs on Unity Visual Scripting (UVS) manipulation tasks, as well as the statistical analysis of the results. Study Overview Seven LLMs are evaluated on eight UVS manipulation tasks spanning four difficulty levels (Easy, Medium, Hard, and Ultra). Each task–model combination is executed five times, yielding 280 executions in total. Correctness is measured with Precision, Recall, and F1-score computed by a rule-based validation pipeline. Statistical analysis applies Kruskal-Wallis tests with Dunn post-hoc comparisons (Bonferroni-corrected), reporting η² as the omnibus effect size and rank-biserial r for pairwise comparisons. Models evaluated: Anthropic Claude 4.5 Sonnet Google Gemini 2.5 Pro OpenAI GPT-5.2 DeepSeek R1 (70b) Google Gemma3 (27b) OpenAI GPT-OSS (120b) Meta LLaMA 3.2 (3b) Tasks by difficulty level: EASY: 01_easy_rotate_naming 02_easy_rotate_broken_wire MEDIUM: 03_medium_movement_speed_dt_broken_wire 04_medium_movement_axis_diag_normalize HARD: 05_hard_rotate_invert_and_fix_names_plus_movement_axes 06_hard_repair_broken_refs_restore_movement_rotation ULTRA: 07_ultra_generate_rotation 08_ultra_generate_movement_rotation Repository Structure ├── README.md ← This file├── LOCAL_LLM_SETUP.md ← Local inference environment used │ ├── 01_Experiment/ ← Experiment runner, tasks, outputs, and validator │ ├── tasks/ ← Benchmark task definitions (8 tasks) │ ├── outputs/ ← Pre-collected LLM outputs (280 executions) │ └── uvs_validator/ ← Rule-based validation │ ├── parsers/ ← UVS asset and pack parsers │ ├── tokenizers/ ← Unit tokenizer │ ├── utils/ ← Graph analysis utilities │ └── validators/ ← Rule checks (basic and advanced) │ └── 02_StatisticalAnalysis/ ← R analysis scripts and data ├── executions.csv ← Scored executions (280 rows; input to R) ├── kruskal_wallis.R ← Full statistical analysis script └── kruskal_results.txt ← Output produced by kruskal_wallis.R License This replication package is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0) (https://creativecommons.org/licenses/by/4.0/). You are free to share and adapt the materials for any purpose, provided appropriate credit is given.