Artifact for "Prompt Pipelines as Engineered Artifacts: Regression-Testing a Multi-Agent Threat-Modeling Pipeline"

Abstract

Replication and verification material for the PROMPTOPS 2026 paper "Prompt Pipelines asEngineered Artifacts: Regression-Testing a Multi-Agent Threat-Modeling Pipeline",presented at the 1st International Workshop on PromptOps and Vibe Coding, co-located withthe 41st IEEE/ACM International Conference on Automated Software Engineering (ASE 2026),Munich, 12-16 October 2026. The paper reports engineering experience from treating a multi-agent STRIDE threat-modelingpipeline as a set of versioned, regression-tested prompt artifacts: modular role prompts,typed inter-stage contracts, pipeline hyperparameters promoted out of code into versionedconfiguration, and a fixed corpus of 24 system architectures with 745 expert-curatedreference threats acting as a regression oracle with cost-aware metrics. This deposit contains: - 24 system architectures across five domains, normalised to a unified YAML schema, and their 745 expert-curated reference threats;- raw output for 8 pipeline variants across all 24 architectures (192 runs), including per-agent and per-stage run metadata;- the aggregate evaluation data behind Table 1 of the paper;- the prompt artifacts: category-agent, retrieval-query, auditor, corrector and risk-scoring prompts, per variant;- a code-only snapshot of the pipeline framework;- verify/verify_claims.py, which recomputes every quantitative claim in the paper and reports PASS, FAIL, or UNSUPPORTED for each. The verification script is the point of the deposit. It currently reports 29 passed, 1failed and 2 unsupported, and that result is intentional: it exposes two reporting defectsthat the paper's camera-ready text addresses in the open, rather than leaving a reader tofind them. UNSUPPORTED marks a claim the retained archive cannot establish. Two limitations of the archive are load-bearing and are documented in MANIFEST.md ratherthan left to be discovered. Risk scoring was enabled for only 7 of the 24 architectures, sothe risk-calibration column aggregates 6 or 7 cells per variant rather than 24. The trialruns that decided the removal of the chunk-filtered retrieval stage were not retained, andneither were the curated extended reference lists; the aggregate values and the outputs theexpansion was drawn from survive, the promotion decisions do not. Not included: the retrieval knowledge base built over CVE, CWE, MITRE CAPEC and ATT&CK, andOWASP material, whose redistribution terms differ from this deposit's licence.framework/KNOWLEDGE_BASE.md records its contents, the provenance of each source, and thecommand that rebuilds the index. No API credentials are included; build_package.py scansthe assembled package for credentials and personal data and fails if it finds any. Licence: the code is MIT. The architecture descriptions and reference threat lists areaggregated from open repositories of published threat-modeling examples and retain theirupstream terms; per-architecture provenance is in dataset/README.md. Reproducing a pipeline run end to end requires an OpenRouter API key and a rebuiltknowledge base. Note that LLM output varies between runs even at the pinned inferencesettings, so a fresh run will not reproduce the archived outputs exactly. The archive holdsone run per (architecture, variant) cell; this single-run limitation is discussed in thepaper.