Experimental Materials and Dataset - "Literature-Grounded Generation of README Files with Large Language Models: A Dual Human–LLM Evaluation"

Abstract

This is the complete replication package for the ISE 2026 paper "Literature-Grounded Generation of README Files with Large Language Models: A Dual Human–LLM Evaluation". It contains every artifact behind every number reported in the paper: the prompts, the generated README files, the raw human and automated evaluations, and the analysis scripts that turn them into the reported statistics.What the study didREADME-Gen is a structured prompting approach in which the README template is fixed a priori. The template is derived from two empirical studies of README content and from the Theory of Robust API Knowledge (ATRAK), with each section bound to one Knowledge Element. Model: gpt-4.1-mini-2025-04-14.Subjects: twelve GitHub repositories across JavaScript, Python and Bash/Shell, deliberately mixing popular projects (>30k stars) and obscure ones (<300).Input: no repository artifacts are injected into the prompt. The model writes the README from prior knowledge and whatever the provider retrieves at inference time.Evaluation dimensions: structural completeness, ATRAK adherence, and correctness.Evaluation protocols: two independent protocols, a human evaluator and an LLM judge (Claude Fable 5). The judge operates as a tool-using agent that installs each project in a throwaway environment and executes every documented snippet.Baseline: README-AI, run under the same model and the same rubric. Contents 209 files, 8.1 MB, organised in pipeline order: 1_inputs/: the twelve repositories with language, stars, popularity group and Core Functionality ground truth, plus the run manifest.2_prompts/: the exact generation prompt, and the generic, tool-agnostic evaluation and comparison prompts (reusable beyond this study).3_generated_readmes/: 48 generated files. 36 are from README-Gen (three independent runs per repository) and 12 are from README-AI.4_evaluations/: the raw evaluation data. It includes the human evaluator's worksheet (CSV and XLSX), 74 score CSVs, and 48 LLM-judge reasoning files. The reasoning files record, per rule, the commands executed, their output and the verdict.5_comparison/: a secondary artifact demonstrating that the comparison workflow generalises. It produces no number used in the paper.6_analysis/: reproduce.py, which recomputes every statistic, table and figure in the paper directly from 4_evaluations/, and its outputs. Reproducing Install 6_analysis/requirements.txt (pandas, scipy, matplotlib).Run python 6_analysis/reproduce.py. The script reads only 4_evaluations/. No summary spreadsheet is consulted, so a reader can verify the aggregation rather than trust it. Deleting 6_analysis/outputs/ and re-running regenerates every table, figure and statistic identically. Both the human and the automated evaluations are rebuilt from the judges' original outputs. Documentation README.md: maps the folder layout and the file conventions.METHODOLOGY.md: explains how each artifact was produced and states plainly what was not controlled:the judge was run at its then-current version rather than a pinned snapshot;ground truth is the released artifact and default branch, not a pinned commit;no execution timeout was fixed.CLAIMS.md: maps every quantitative claim in the paper to the artifact that produces it. Not included README-Gen's own source code (the CLI/HTTP application) is not part of this package. 2_prompts/generation_prompt.md is the complete record of what determines the tool's output. Regenerating a README requires only that prompt, an OpenAI key and the pinned model.The earlier, superseded round of automated scoring is also not included. 4_evaluations/ is the single source of truth.