Evaluating LLMs for Single-File Modernization of Legacy PHP: Source Modernization versus Behavioral Preservation

Abstract

Replication package for the paper "Evaluating LLMs for Single-File Modernization of Legacy PHP: Source Modernization versus Behavioral Preservation." The study evaluates five large language models (Claude Sonnet 4, Gemini 2.5 Pro, Gemini 2.5 Flash, GPT-5 Codex, and Llama 3.3 70B Instruct) on modernizing 100 WordPress 4.0 PHP files to PHP 8.3 when only the target file is available. Each model receives one complete file and a fixed instruction, and returns one response per file. The rule-based tool Rector is evaluated under the same single-file condition. Two outcomes are measured separately: Behavioral compatibility: 630 hidden PHPUnit checks executed under PHP 8.3, compared against the unchanged originals (no-change baseline: 83 of 100 files compatible; 531 of 630 checks pass). Source modernization: resolution of 511 Rector-detected legacy patterns across 44 PHP-version rule types, grouped as required, deprecated, or optional under PHP 8.3. CONTENTS Dataset: the 100 selected WordPress 4.0 files, the selection script and seed, and Rector findings for the selected files and the full 482-file corpus. Prompt: the modernization prompt used for all LLM runs. Model outputs: raw model responses and extracted migrated files for each model, plus syntax-validity and structural-completeness reports. Rector baseline: the Rector configuration (UP_TO_PHP_83 level set) and the script that generates the rule-based output. Hidden checks: the PHPUnit harness and supporting environment for the 100 files. Results: per-file and model-level compatibility outcomes, the check-level transition matrix (preserved pass, regression, repair, unrepaired failure), regression-cause classification, legacy pattern resolution per model, and statistical analysis outputs. Scripts that regenerate every table in the paper. REPRODUCING THE RESULTSFrom the PHP_Migration directory, run the PowerShell script run_full_evaluation.ps1. It executes all stages in order and regenerates every table in the paper. Exact tool versions (PHP 8.3, Rector 2.1.0, Composer) are pinned in composer.lock and reproducibility.lock.json; verify_reproducibility.ps1 checks the environment. KEY ENTRY POINTS- Full pipeline: PHP_Migration/run_full_evaluation.ps1- LLM modernization: PHP_Migration/LLM_Migration/scripts/migrate.py- Dataset selection: PHP_Migration/dataset/select_optimal_files.py- Legacy pattern resolution: PHP_Migration/LLM_eval/run_evaluation.py- Hidden checks: PHP_Migration/UnitTest/wordPress/- Prompt: prompts.py- No-change baseline outcomes: OriginalTestResults.txt See README.md in the package for the full directory map and a description of every output file.