UnifiedBench: Toward a Unified Dataset for Red-Teaming Language Models

Abstract

Adversarial prompting constitutes one of the principal threats to deployed Large Language Models, yet the evaluation of alignment robustness against such attacks is hindered by the fragmentation of existing benchmark datasets. AdvBench, HarmBench, and JailbreakBench — the three most adopted adversarial prompt datasets — adopt mutually incompatible categorical schemes, which prevents cross-dataset comparison of category-level vulnerability. We address this gap through four contributions: (i) a unified taxonomy of harmful behaviors anchored in the NIST AI 100-2 framework, comprising seven categories that decompose the Misuse vulnerability class; (ii) UnifiedBench, a consolidated dataset of 810 adversarial prompts derived from the three source datasets and categorized under this taxonomy; (iii) a query-only evaluation pipeline combining an automated judge (LlamaGuard-3-1B) with human reviewers to measure per-category Attack Success Rate (ASR); and (iv) a comparative analysis of two instruction-tuned models under direct harmful prompting. Our findings reveal current alignment procedures are comparatively less effective at preserving deployment-specific operational constraints than at preventing conventionally harmful content.