Replication package for "Production-Oriented Evaluation and Testing of LLM-Based Agents: A Systematic Literature Review"

Abstract

Replication package for the systematic literature review "Production-Oriented Evaluation and Testing of LLM-Based Agents: A Systematic Literature Review" (revised version, submitted to Information and Software Technology; first submitted as "Evaluation and Testing of LLM-Based Agents in Production: A Systematic Literature Review"). The review follows Kitchenham–Charters guidelines with PRISMA 2020 reporting. The main search covers Scopus, Web of Science, IEEE Xplore and the ACM Digital Library up to 24 March 2026; with citation tracking it yields a review corpus of 88 primary studies (74 from the main search, 14 from citation tracking). A search extension to 25 September 2026 with the same query and criteria yields 314 further studies, reported as a separate update analysis. A Scopus sensitivity search with broader production-oriented terms (300-record random sample) estimates the recall of the query. Version 2.0.0 accompanies the revised article. Version 1.0.0 (DOI 10.5281/zenodo.20583928) accompanies the article as first submitted and is preserved unchanged in the folder 2026-03/. Contents: corpus/, extraction/, risk-of-bias/ — the review corpus of 88: included and excluded studies with screening decisions, dual data extraction (Reviewer 1, Reviewer 2 and arbiter-resolved; 27 analytical fields) and three-domain quality assessment. 2026-03/ — the March 2026 round exactly as released in v1.0.0 (80 studies, agreement reports, audits). 2026-09/ — the September 2026 round: ACM Digital Library and update searches (full-text records R001–R670), Scopus sensitivity search, extraction and quality assessment of 335 studies, agreement reports, and the single-label recode of automated-testing with its held-out validation. primary-studies.bib (88) and 2026-09/primary-studies.bib (335). reproduce.ipynb — a self-contained notebook with 595 automated checks that reproduce every statistic of the article computed from the per-record data (Section 2: review corpus; Section 3: September round; Appendix A: the 2026-03 round as first submitted). README.md (structure, data dictionary, errata), CHANGELOG.md, methodology.md, statistical-analyses.md, prisma-2020-checklist.md, kitchenham-2007-compliance.md, requirements.txt, LICENSE. To reproduce: install the dependencies (pip install -r requirements.txt) and run reproduce.ipynb top to bottom. Expected output: 595 checks passed, 0 failed. Licensed under CC BY 4.0.