replication package
Abstract
# Replication package ## Intelligent Agile Adoption Framework — archival demonstration (Section 5) Khaled Ismail · ORCID 0000-0002-0499-5805 ### What this tests Proposition 2 of the manuscript is separated into two claims and both are tested here. * **P2a** — the dominant component of squared forecast error in human Agile estimation is dispersion, not bias. **Supported.** * **P2b** — predictive inference reduces that dispersion component. **Not supported** on this corpus, in a result consistent with independent replications (Tawosi et al., 2023, 2024). Pooled human forecast error: SD 2.358 log-units, mean +0.071; the central 80% of issues are realized at 0.05x to 13.5x their planned duration. A third analysis (`p2\_conformal.py`) compares interval calibration methods on identical predictions. Quantile regression attains **64.3%** coverage on a nominal 80% interval; locally adaptive split conformal attains **76.6%**, within 5 points of nominal in 21 of 24 projects (Wilcoxon p = 5.2e-6, rank-biserial 0.91), at a 28% cost in interval width. The severe under-coverage is therefore largely an estimator artifact rather than an irreducible property of the data; the residual 3.4-point shortfall (p = 0.039) indicates mild non-exchangeability, i.e. process drift. ### Data TAWOS — a versatile dataset of agile open source software projects (Tawosi, Al-Subaihin, Moussa \& Sarro, MSR 2022), DOI 10.1145/3524842.3528029. Dataset DOI 10.5522/04/21308124, Apache License 2.0. 458,232 issues, 39 projects, 12 public Jira repositories. The dataset ships as a MySQL dump. `extract.py` stream-parses it directly and requires no database server. ### Pipeline ``` python3 extract.py # TAWOS.sql -> issues.csv (458,232 rows) python3 p2\_analysis.py primary # total-effort outcome, revised estimates excluded python3 p2\_analysis.py sens\_resolution # resolution-time sensitivity python3 p2\_conformal.py # conformal vs quantile interval calibration python3 delegation\_rule.py # regenerates Table 3, Table 4 and Table S6 python3 figs.py # regenerates Figures 1, 2, 3 and S1 ``` ### Analysed sample |Specification|Projects|Predictions| |-|-|-| |primary (Total\_Effort\_Minutes)|25|31,356| |sensitivity (Resolution\_Time\_Minutes)|32|43,536| Inclusion: story point > 0; outcome > 0; story point not revised after estimation; at least 150 prior issues before the first prediction; at least 60 evaluable predictions per project. ### Method Error is taken in log space, e = ln(actual) − ln(predicted), giving the exact decomposition MSLE = mean(e)² + var(e) into a bias and a dispersion term. Three forecasters are evaluated strictly out of sample under a time-ordered walk-forward within each project: * **H** human — story point × historical minutes-per-point rate (velocity planning) * **R** reference class — historical median duration of prior same-type issues * **M** model — gradient-boosted quantile regression at the 0.10 / 0.50 / 0.90 quantiles Inference across projects uses Wilcoxon signed-rank tests, 10,000-sample bootstrap confidence intervals, and matched-pairs rank-biserial effect sizes. The unit of analysis is the project (n = 25), not the issue. ### Known limitations * The outcome is a **duration proxy** derived from issue state transitions, not logged effort. Only 900 issues in the corpus carry logged work, too few to analyse. * The corpus contains **none of the Layer 1 telemetry** the framework specifies — no CI outcomes, work-in-progress levels, dependency depth, review latency or team composition history. The P2b test is therefore a joint test of P1 and P2b, and a null is what P1 predicts under thin telemetry. * The model arm is one model class with one feature set. ### Environment Python 3.13 · numpy 2.4 · pandas 3.0 · scipy 1.17 · scikit-learn 1.8 · matplotlib. Random seed 20260921 for all bootstrap resampling. ### Figure palette The two-colour categorical palette (#2E6FA8, #C2681A) was validated for colour-vision deficiency separation, lightness band, chroma floor and surface contrast before use.
