Skip to content

SoK: Rethinking Jailbreaking in the Era of Agentic AI: Attacks, Defenses, and Practical Consideration

Sep 2026 · 0 citations
Computer Science

TL;DR

A Systematization of Knowledge (SoK) that reframes jailbreak security around the full agentic execution pipeline, and develops unified taxonomies of attacks and defenses spanning user interaction, planning and reasoning, memory, tool use, and inter-agent communication, introduces a security--utility--efficiency evaluation framework.

Abstract

Large language models (LLMs) are rapidly evolving from conversational assistants into agentic AI systems that reason, plan, invoke tools, maintain persistent memory, communicate with other agents, and execute multi-step tasks. At the same time, modern models exhibit substantially stronger native safety alignment than earlier generations on which many jailbreak attacks and defenses were originally studied. This shift raises a fundamental question: \textit{which established jailbreak-security findings remain valid in the era of modern LLMs and agentic AI?} We address this question through a Systematization of Knowledge (SoK) that reframes jailbreak security around the full agentic execution pipeline. We develop unified taxonomies of attacks and defenses spanning user interaction, planning and reasoning, memory, tool use, and inter-agent communication, and introduce a security--utility--efficiency evaluation framework that separates native harmful-prompt safety, adversarial jailbreak robustness, and agent-level security outcomes. We further conduct a controlled empirical study of representative attacks and defenses within a common agentic framework. Our results reveal three important gaps. First, strong native alignment does not imply robustness to adversarial jailbreaks. Second, defense effectiveness is highly model-, attack-, and component-dependent and can come at substantial cost in over-refusal, utility, and latency. Third, low final-response attack success can mask severe intermediate compromise: planning, memory, and tool interactions may remain unsafe even when the final response is successfully filtered. These findings motivate a shift from response-centric jailbreak defense toward cross-layer, execution-aware security that protects agent state, component transitions, and external actions while preserving practical utility and efficiency.

View source

Similar papers

Preprint Aug 2026

A Self-Evolving Multi-Agent Framework Defense against LLM Jailbreak Attacks

This work proposes a self-evolving test-time defense built around a persistent, cross-interaction rule memory that substantially reduces attack success rates while preserving benign utility, remains robust under an adaptive composite-wrapper attack, and does not increase over-refusal as the memory grows.

Tongshen Hu, Bryan Hooi · 0 citations

Jailbreaking Jailbreaks: A Proactive Defense for LLMs

P RO A CT represents an orthogonal defense strategy that serves as an additional guardrail to enhance LLM safety against the most effective attacks.

Wei-Liang Zhao, Daniel Ben-Levi, Jinjun Peng et al. · 1 citation

AIOracles: Modeling Agentic AI Security *

A formal framework for security in agentic AI is developed, built around an ideal predicate ϕ that decides, for a given ( prompt, context, result ) triple, whether the agent’s behaviour is acceptable.

F. Durak, Tadayoshi Kohno, Franziska Roesner · 0 citations
#artificial intelligence Preprint Oct 2026

Benchmarking Jailbreak Guardrails for Embodied Agents

Embodied agents powered by large language models and vision-language models are increasingly deployed in physical environments, but jailbreak attacks can induce these agents to perform physically harmful actions. A growing number of guardrail methods have been proposed to intercept dangerous behavior before it is execu...

Xun-Guang Wang, Qing-Yue Wang, Yu-Guang Zhou et al. · 0 citations
Preprint Aug 2026

SysEvolve: An AI-native, safe, autonomous adversarial attack-defense co-evolutionary system

The rapid advancement of large language models (LLMs) has created a growing asymmetry in cybersecurity, where attack accelerates toward autonomous execution while defense remains predominantly human-intensive. Despite substantial prior work across cyber ranges, AI-driven attack, and AI-driven defense, this asymmetry pe...

Yuhan Meng, Shao-Fei Li, Jiong-Hao Huang et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.