R+R: TTP-WorkBench: A Reproducibility Framework and Benchmark for TTP Extraction from Cyber Threat Reports
Abstract
TTP-WorkBench Artifact TTP-WorkBench is a framework and benchmark for executing and comparing ten tools from prior work that extract MITRE ATT&CK tactics, techniques, and procedures from cyber threat reports. The artifact contains standardized tool adapters, container build specifications, curated prior-work data, and 281 new threat reports for benchmarking the tools. Provided scripts automate the container builds, dataset compilation, functionality tests, reproduction experiments, and capacity-aware analysis. The artifact documentation gives a setup procedure and three evaluation exercises to demonstrate that the artifact is functional and reproducible. Paper Abstract In cybersecurity operations, threat reports are a valuable resource to understand emerging threats, with MITRE ATT&CK serving as a common framework for interpreting the attacker’s tactics, techniques, and procedures (TTPs) described in the reports. Manually inferring TTPs from textual reports is difficult and time-consuming, which has motivated dozens of prior studies in automated extraction and labeling approaches. However, there has been no comprehensive comparison across tools, perhaps because many open-source implementations are difficult to run, lack experimental artifacts, or do not perform well on new data. While others have pointed out these systemic obstacles to reproducibility in the broader security research community, we practically analyze and address them in the field of automated TTP extraction. In this paper, we first analyze the TTP extraction literature through the lens of artifact reusability and then develop a reproducibility framework that directly addresses key challenges discovered while reviewing previous work. Furthermore, we create a benchmark dataset that curates existing data and adds 281 new threat reports with 4,732 TTP ground-truth labels. We demonstrate the usefulness of our reproducibility framework and benchmark dataset by adapting ten previously proposed tools. We then use the framework to reproduce key experiments published in prior work and test the tools on never-before-seen data. Our work advances the automated TTP extraction field by facilitating the reproducibility of past research and by making it easier to systematically compare future TTP extraction tools on a new benchmark dataset. Release - Version: `acsac26-ae-v1` - Publication date: `2026-09-20` - Zenodo DOI: `10.5281/zenodo.22839412` - GitHub release/tag: `https://github.com/stfran/ttp-workbench/tree/acsac26-ae-v1` - Git commit: `512b2749fe30612dfcc021cb962d4e7901b3f69f` Files and Integrity File Purpose Size (Bytes) Sha1 ttp-workbench-artifact.zip Artifact evaluation package and clean TTP-WorkBench source snapshot 4900909 333f2b5b0cb10732c81832aaf581bef3fd2a246f models.zip Models and other large reproduction assets used by the installer 37714873848 c905bf913b0b915ae1f324b69987fcb58de5524a` Download both ZIP files into the same directory. Extract `ttp-workbench-artifact.zip`; its `install.sh` locates the adjacent `models.zip`. Setup, evaluation, expected results, and teardown instructions are in `README.txt` and `use.txt` inside the extracted artifact directory. Licenses Project-authored software is released under the Apache License 2.0. Project-authored data is released under the Creative Commons Attribution-ShareAlike 4.0 International license (CC BY-SA 4.0). Third-party software, models, datasets, report text, and MITRE ATT&CK material remain under their respective licenses or terms, as documented inside the artifact. `models.zip` contains checkpoints based on Llama 3.1 and the archive retains the Llama 3.1 Community License and required attribution notice. Built with Llama.
