Reproducibility Package for "A Governance Architecture for AI-Assisted Validation of Construction Cost Knowledge: Combining Domain Rules with LLM Semantic Analysis"

Abstract

This deposit contains the reproducibility materials for the manuscript "A Governance Architecture for AI-Assisted Validation of Construction Cost Knowledge: Combining Domain Rules with LLM Semantic Analysis." The manuscript presents a deployed tender management system that validates Bill of Quantities (BOQ) pricing formulas at portfolio scale by combining trade-specific deterministic policy checks with large language model (LLM) semantic analysis under a four-layer hallucination-mitigation architecture (schema enforcement, evidence binding, deterministic override, and read-only AI). The evaluation includes 17 construction tenders across Morocco and Ivory Coast, representing approximately 1.1 billion monetary units of portfolio value, and 100 expert-labelled formulas with an inter-rater agreement of Cohen's κ = 0.96. Package contents (10 files): README.docx — Package overview, file listing, sanitization policy, and citation guidance. CHANGELOG.txt — Version history describing updates to each release. Reproducibility_Supplement.pdf — Dataset definitions, key statistics, leakage-prevention protocol, baseline model specifications, statistical analysis details, the complete verbatim LLM prompt template, and a worked input-output JSON trace for the REV-002 example. LLM_Prompt_Template_Full.docx — Complete verbatim system and user-message templates used with the Azure OpenAI gpt-5.1-codex-mini deployment. Results.xlsx — Headline manuscript results, including Expert Metrics, Confusion Matrices, Cross-Region F1, Risk Distribution, and the Data Dictionary. scripts/verify_from_deposit.py — Verification script that recomputes the manuscript's expert-sample confusion matrices, overall and per-region F1 scores, and portfolio risk distribution directly from the deposited data, printing the recomputed and reported values side by side. Data/Expert_Validation_Sample.xlsx — The 100 expert-labelled formula records, including the associated construction trade for each formula. Data/Validation_Mode_Predictions.xlsx — Per-formula predictions for the four validation configurations (Hybrid, Rules-only, AI-only, and AI-with-flags). Data/Baseline_Predictions.xlsx — Outputs for the six baseline models (LR-S2, XGB-S2, RF-S2, Isolation Forest, z-score, and TF-IDF+LR), together with the baseline ranking and fold composition. Data/Statistical_Tests_Frozen.xlsx — Official statistical-test artefacts, including McNemar's test, i.i.d. and TenderId-cluster bootstrap confidence intervals, per-fold stability, and inter-rater agreement. Sanitization. Item descriptions, unit prices, and AI-generated narrative text have been removed, while FormulaId and TenderId have been pseudonymized. Component percentages, construction trades, expert labels, and risk classifications—the variables used in the manuscript's analyses—remain intact. Consequently, all quantitative results reported in the manuscript can be reproduced from the released artefacts. Version notes. This release aligns per-formula predictions across validation modes using the composite key (TenderId, FormulaId), adds the per-formula Trade column, and includes the verification script. See CHANGELOG.txt for the complete revision history. Raw BOQ text and original monetary values are not publicly released because they contain commercially confidential contractor pricing from active projects. Researchers requiring access to additional anonymized data may contact the corresponding author. Version 5 (September 2026) accompanies the revised manuscript. It adds the trade-label audit workbook (Data/Trade_Label_Audit.xlsx, 146 screened candidates and 80 controls with the senior estimator's verdicts) and its screening script, the three-model replication outputs (Data/second-model/, per formula per arm per configuration), a CHANGELOG.txt describing every change since v4, and corrected Region and component columns in the expert-sample workbook; no reported metric changed. README.docx lists the contents; scripts/verify_from_deposit.py recomputes every headline number from the deposited files.