Automated Attack Trace Generation: Investigating Coverage and Cleanup Resilience

Abstract

Attack trace datasets reflecting each organization's network and device configuration are essential for evaluating detection rules and incident response procedures, yet generalized datasets abstract such configurations away. To address this, we proposed a method that automatically selects, executes, and collects traces from feasible MITRE ATT&CK techniques within an operator-prepared virtual environment. However, its prototype collected only file differences and the Target's standard event logs, and did not account for volatile or post-cleanup traces. In this paper, we extend the trace collection mechanism with file-I/O API hooks and router-side communication log collection, and systematically evaluate the 747 Atomic Red Team techniques applicable to our experimental environment, quantifying per tactic and per log source the trace evidence captured before and after attacker cleanup. Approximately 86% of techniques were executable to completion, and traces remained after cleanup for about 76% of the executable techniques (rising to 83% under full-traffic logging). The resulting per-tactic, per-source coverage characterization provides a baseline for optimizing organizational monitoring configurations and prioritizing incident response.