PromptLeaks: Assessing Data Leakage in Guardrails and Automated Jailbreak Tools
Abstract
Large language models (LLMs) have become ubiquitous, and with their adoption comes a range of ethical and security risks that has made alignment, and resistance to jailbreaking, a central concern. Guardrails are a widely adopted defense against such jailbreaks, yet their reported effectiveness is only as reliable as the data on which they are evaluated. This paper identifies a largely unexamined source of inflated performance: data leakage between the datasets used to seed offensive jailbreak frameworks and those used to train defensive guardrails. As both are derived from human-authored jailbreak prompt corpora, a guardrail may be evaluated on instances present in its own training data, in which case high detection rates reflect memorization rather than a generalized alignment policy. We quantify this contamination, which we term PromptLeaks, using a deterministic syntactic comparison of prompt datasets adapted from bioinformatics. Across three open-source red-teaming frameworks and five open-source guardrail datasets we observe severe leakage, with one test-prompt dataset almost entirely contained within the guardrail training corpora. Guardrail jailbreak detection rate diminishes on prompts that do not overlap the training corpus, indicating that current guardrails rely substantially on memorization rather than on generalizing an alignment policy.