Replication Package — Chapter 7: "From Diagnosis to Prescription: Retrospective Evaluation of Prescriptive Implications" (PhD Thesis: From Novice to Proficient)

Abstract

Replication package for Chapter 7 of the PhD thesis "From Novice to Proficient: Programming Skill, Design Quality, and Abstraction in Scratch — from Diagnosis to Empirically Supported Prescription" Author: Ricardo Hidalgo AragónSupervisors: Dr. Jesús M. González Barahona, Dr. Gregorio Robles MartínezInstitution: Universidad Rey Juan Carlos (URJC), Escuela Internacional de DoctoradoDoctoral Programme: Doctorado en Tecnologías de la Información y las ComunicacionesYear: 2026 Chapter 7: From Diagnosis to Prescription (RQ-G4) Chapter 7 closes the loop of the thesis: after diagnosing programming skill (Ch4, CEFR framework, B2 bottleneck), code quality (Ch5, CT–smell mapping, 95.9% size mediation), and the abstraction gap (Ch6, cloning vs. custom blocks), it evaluates the prescriptions implied by those diagnoses through three retrospective observational and evaluative analyses on the EMSE Scratch corpus (1.64M projects, 729K authors). No intervention is implemented on learners; the design is observational, so results are reported as associations conditional on measured covariates: RQ-G4.1 — Abstraction threshold (Propensity Score Matching): Is custom-block adoption associated with lower clone density? After PSM matching on project size and CT score (358,219 matched pairs), custom-block projects show ~28% lower clone density (median 0.75 vs 1.05, r=0.58, p≈0) — the prescription receives retrospective observational support; the raw null result was entirely driven by the size confound. A complementary depth analysis (parameterized blocks as treatment) finds no consistent additional association beyond adoption. RQ-G4.2 — Context-aware linter (Binary classification + McNemar): Does taxonomy-weighted scoring of LitterBox reports (w_stable=2.0, w_sensitive=0.5, τ=0.5) improve the pedagogical signal? At a comparable threshold the weighted linter reaches precision 0.84 / recall 0.85 (F1 0.84), clearing 22 of the 66 justified patterns the raw detector flags (TN 0→22); McNemar χ²=20.05, p=7.6e-6. Neither method beats the degenerate flag-everything F1 (0.89 at τ=0), so the prescription receives qualified evaluative support as precision at matched recall — the weighted score dominates raw counting in precision at every recall level below ≈0.90 (e.g. 0.90 vs 0.80 at recall 0.60). A purely suppressive 3-rule variant (precision 0.94, recall 0.63, F1 0.76) is explored in the grid search and rejected. Supplementary PR curve + 40-config grid search explores the full operating space. RQ-G4.3 — B2 bottleneck and stagnation (Kaplan-Meier survival analysis): Do learners who start at higher quality stagnate more? KM curves clearly separate: Beginner 61.2% event rate (median 3 projects), Intermediate 39.7% (median 4), Advanced 26.5% (median 8). All pairwise log-rank p≈0. The gradient is graded and statistically robust, providing retrospective longitudinal support for the Cap. 4 prescription (higher entry-level proficiency predicts slower progression); it complements, but does not directly validate, the CEFR-specific B2 bottleneck as a longitudinal phenomenon. Repository Contents code/ — 14 Python scripts (steps 00–12, with two variants of step 07), fully ordered pipeline (run_pipeline.sh):- 00: Build author/remix trajectories from EMSE parquet- 01: Extract per-project abstraction (optional, requires MongoDB)- 02: RQ-G4.1 primary (within-author before/after)- 03: RQ-G4.2 primary (context-aware linter classification)- 04: RQ-G4.3 primary (mastery/CEFR progression rates)- 05: Consolidated statistical tests (BH-FDR correction)- 06: Generate all figures- 07: RQ-G4.1 supplement (PSM matching) / 07: RQ-G4.1 supplement (PSM depth)- 08: RQ-G4.3 supplement (Kaplan-Meier survival)- 09: RQ-G4.2 supplement (PR curve + grid search)- 10: Triangulation analysis- 11: Triangulation figures- 12: Figure T3b data/ — Intermediate parquets (trajectories, abstraction, labels):- trajectories_author.parquet (1.64M rows)- trajectories_remix.parquet (remix chain lineages)- abstraction_perproject.parquet (CB + clone metrics)- t1_labels_emse.csv / t1_labels_emse_summary.json results/ — All analysis outputs (JSON, CSV, LaTeX):- Per-RQ result files (rqg41_results.json, rqg42_metrics.csv, rqg43_results.json, etc.)- PSM balance + results (rqg41_psm_results.json, rqg41_psm_balance.csv)- Survival analysis (rqg43_survival.json, rqg43_survival_km.csv)- PR curve (rqg42_pr_curve.csv)- Consolidated statistical tests (statistical_tests.json, statistical_summary.csv)- Triangulation outputs (triang_T1_*.csv, triang_T2_q4.json, triang_T3_*.parquet)- Pipeline logs for every step figures/ — 14 publication-quality figures (PDF + PNG at 300 DPI):- Violin plots (before/after CB adoption)- SMD balance plot + matched comparison (PSM)- Precision/Recall/F1 grouped bar + per-smell F1- Full PR curve with operating points- Mastery/CEFR progression bars- Mastery transition heatmap- Kaplan-Meier survival curves- Effect size forest plot- Triangulation figures (context, detail, panel, T3b) Documentation:- README.md — full chapter overview, sub-studies, key results, verification checklist- DATA.md — data source table and join keys- PIPELINE.md — step-by-step pipeline description with dependencies- RESULTS.md — result interpretation guide- requirements.txt — Python dependencies (numpy, pandas, scipy, statsmodels, sklearn, matplotlib, seaborn) How to Reproduce bash# Full pipeline from scratch (steps 0–10; step 01 requires MongoDB) ./code/run_pipeline.sh # Skip MongoDB step (RQ-G4.1 uses custom_block_usage proxy instead) ./code/run_pipeline.sh --from 02 # Run only supplement analyses + regenerate figures ./code/run_pipeline.sh --from 7 # Quick smoke test on 2,346-row sample (~2 min) ./code/run_pipeline.sh --smoke Python 3.12+ with dependencies from requirements.txt. Related Publications Hidalgo-Aragón, R., González-Barahona, J.M., Robles, G. (2026). "A CEFR-Inspired Classification Framework with Fuzzy C-Means To Automate Assessment of Programming Skills in Scratch." CSEDU 2026 — DOI: 10.5220/0014691000004021 (Chapter 4) Hidalgo-Aragón, R., González-Barahona, J.M., Robles, G. (2026). "How Computational Thinking Skills Influence Design Quality in Scratch: An Empirical Analysis of Code Smells" Empirical Software Engineering — under review. (Chapter 5) Hidalgo-Aragón, R., González-Barahona, J.M., Robles, G. (2026). "Code Cloning and Procedural Abstraction in Scratch Projects" — under review. (Chapter 6) Corrigendum & provenance note (2026-09-10) This version adds two provenance audits (scripts t1_diagnostic.py and table4_diagnostic.py) and a corrigendum note. It does not change the Chapter 7 pipeline or its conclusions. Chapter 4 centroids ("Table 4"). The authoritative centroid artifact is fcm_model.pkl = ordered_centroids.csv; re-running the documented FCM (k=6, m=1.5, epsilon=1e-5, seed=42) reproduces it exactly. Those centroids place the abstraction_lag maximum at B2 (+0.528), consistent with this chapter. The centroid table printed as "Table 4" in the CSEDU 2026 manuscript is not reproducible from the package and is superseded by these values (it would place the maximum at A1, +0.203). CEFR labels. Chapter 7 uses data/t1_labels_emse.csv (FCM applied to the EMSE corpus; 100% coverage). skillvae/artifacts/t1_labels.csv overlaps the EMSE corpus in only 1.04% of project IDs and was not used. Kruskal-Wallis. Correct statistic for abstraction_lag ~ cefr_band: H = 169,490.31. The value H = 106,295.94 corresponds to intra_clone_density ~ cefr_band. See results/CORRIGENDUM_2026-09-10.md, results/table4_provenance.md, results/t1_provenance.md, results/table4_rerun_centroids.csv and results/model_inventory.csv.