Reproducibility Package for "From Generative AI to Auditable Engineering: A Human-in-the-Loop Assurance Framework for Engineering Artifacts in the V-Model"

Abstract

This repository contains the reproducibility package for the secondary empirical analyses reported in the manuscript “From Generative AI to Auditable Engineering: A Human-in-the-Loop Assurance Framework for Engineering Artifacts in the V-Model – Case of an Intelligent UAV Payload”, by Ali Kamel Issmael Junior and José Vicente Calvano. The package supports the external empirical triangulation used in the study to examine selected assumptions underlying the proposed assurance framework. It does not constitute a complete empirical validation of the framework, its gates, the AAL materiality scale, the Consequential Semantic Unit (CSU/USC) construct, or the causal hypotheses proposed in the manuscript. Those elements require prospective evaluation. Two independent public datasets are reanalyzed: LLM-Generated Software Requirements from GitHub Issues, Zenodo record 19520570, version 2.1.1. The analysis uses the human-validation subset evaluated by five independent reviewers and reproduces inter-rater agreement statistics, LLM–human associations, LLM–human score differences, and correlations between textual length and human assessments of Unambiguity, Verifiability, and Singularity. Microsoft coderec_programming_states telemetry dataset, associated with the study Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming. The analysis examines Copilot suggestion acceptance and rejection events, subsequent editing of accepted suggestions, suggestion length, model confidence, and time spent in Copilot-specific interaction states. A binomial Generalized Estimating Equations (GEE) model with robust covariance and clustering by user is used to assess associations between suggestion characteristics and subsequent editing. The package includes: Python scripts for both reanalyses; a README documenting all operational choices; a Python environment and dependency specification; a fixed project seed (20260906); derived CSV and JSON result files; sensitivity analyses for alternative definitions of subsequent editing; GEE coefficient tables and model summaries; user-level Copilot supervision-time estimates. The original Zenodo and Microsoft datasets are not redistributed in this package. Users must obtain them from their respective public repositories and provide the ZIP files as inputs to the supplied scripts. The analyses are deterministic and use no stochastic estimator or resampling in the reported results. The project seed is included to ensure reproducibility of possible future extensions involving resampling. The package was developed to improve transparency and auditability of the empirical results reported in the associated manuscript. In particular, it documents analytical decisions such as the definition of subsequent editing, sensitivity thresholds, complete-case treatment of missing model-confidence observations, the specification of the GEE model, and the calculation of confidence intervals for Copilot-specific supervision time.