Agent-Mediated Phase Transitions in Software Delivery: Search, Screening and Coding Data

Abstract

Single-coder extraction, checked against an independent coder on a sample. First-pass screeningand coding were performed by a single coder. The 10% sample (dual_screen_sample = YES, n = 19,drawn with random.Random(7410238)) was screened twice, on two separate checks with differentepistemic value: Intra-coder test–retest — the same coder, in a fresh session and blind to its own first decision, reproduced 18 of 19 decisions: 94.7% agreement, Cohen's kappa = 0.883. This measures stability under repetition and cannot detect a bias shared by both passes. Inter-coder agreement — a second, independent coder (the author), blinded to the first pass, agreed on 14 of 19: 73.7% agreement, Cohen's kappa = 0.463 (moderate; SE 0.206; 95% CI 0.06–0.87). Five records were decided differently: A024, A034, A052 and A070 (first coder excluded, second coder admitted) and A150 (first coder admitted, second coder excluded). The second coder's decisions were adopted, so the included set is 65 of 194 in-window records; the four admitted records were coded on all eight facets. Both statistics are reported separately in 08_dual_screening.json, which marks inter_rater_agreement as the primary statistic and sets WARNING to null. 08_human_coder_decisions.csv records the second coder's decisions together with the first-pass decision and rule for each sampled record, so the statistics stay reproducible now that the adopted decisions sit in the corpus. The 19 sampled records carry a decision agreed by two coders; the remaining 175 carry the first coder's decision alone. The corpus is therefore screened to two standards, an asymmetry the manuscript states in Section 3.8 and carries into Section 6.2 as a threat to validity. Grey literature only, in this strand. arXiv is coded as grey literature and is never pooled withpeer-reviewed evidence. The eight peer-reviewed databases in Table 4 are pending institutional access; theirresults must be appended before the frequency tables are treated as the study's final result. Screening aid, not decision. 02_screening_sheet.csv is a keyword triage aid. The decisions in03_screening_decisions.csv were made on the abstract text by the coder; the aid's proposed bucket is retained only for transparency about how attention was allocated. Time window. Four records pre-date January 2023 and are excluded by the registered window. Terminology instability. The constructive threat registered in the protocol (Section VII) is observable in thedata: 04_analysis_RQ4_seed_crosswalk.csv shows Section V phenomena appearing under unrelated vocabulary, which is why single-term booleans under-recall this literature. v4 adopts the second coder's decisions into the corpus. The 10% sample was screened by the author as second coder; agreement with the first-pass decisions was 14 of 19 records (73.7%, Cohen's kappa = 0.463, moderate), against 18 of 19 (94.7%, kappa = 0.883) for the first coder's own blinded re-screen. The four records the second coder admitted (A024, A034, A052, A070) were coded on all eight facets and A150 was removed, so the corpus is now 65 included studies of 194; every count and table was recomputed. 08_human_coder_decisions.csv now also carries the first-pass decision and rule for each sampled record, so the reliability statistics stay reproducible after adoption. The manuscript reports both figures and treats their divergence as the study's principal limitation. Version 5 (28 September 2026) adds the peer-reviewed strand, its harvest, the second-coder measurement and the JSS revision material. Peer-reviewed strand (Amendment 3). The registered string was executed on open cross-publisher indexes — OpenAlex, with Crossref for the publisher-level verification — because the eight subscription databases named in the protocol are not accessible to the author. New here: data/strand_p/ (28 candidates with their strand label, 19 screening decisions each with its rule and a one-line rationale, the funnel and counts), data/open_harvest/ (the verified records and the unmodified raw grid responses), and the harvest, verification and figure code. The funnel: 147 records returned → 128 verified against all three blocks of the string → 42 peer-review types → minus 9 Zenodo deposits, 12 duplicates and 2 records already in the grey strand → 19 screened → 17 excluded (E1 15, E3 1, RQ1NEG 1) → 2 included, coded separately and never pooled with the grey strand. Second-coder measurement (Amendment 4). second_coder/ carries both packs as issued, the sample manifests, the independent coder's returned files and the scoring scripts; data/08b_second_coder_reliability.json reports the measured round — 40 of 40 screening decisions, 20 of 20 exclusion rules and 7 of 7 boundary classifications agreed, reported as raw agreement with a Clopper-Pearson one-sided 95% lower bound of 92.8% (kappa is degenerate at perfect agreement). The first, uncalibrated pilot round (47.5% agreement, kappa = -0.050) is retained in data/08b_pilot_v1_NOT_REPORTED.json as evidence about the instrument; it is not a reliability figure for the corpus. The independent coder is identified as "Coder 02" throughout, and the answer keys and the private correspondence are deliberately not included. Also in this version: the two-strand screening-flow figure with the numbers it draws on (figures/), the two-strand counts (data/flow_counts.json), the citation map for the revision (data/refs_jss_map.json), the SEGRESS alignment and amendment records (protocol/), 00_ADDENDUM_v5.md describing every addition, and MANIFEST_sha256.txt giving the SHA-256 digest of every file.