Oct 2026· Proceedings of the ACM on Software Engineering· Vol 3, pp. 221 - 245· 0 citations· 95 references
TL;DR
Datura, an automated red teaming testing framework that exposes safety vulnerabilities through chained tool manipulation, is proposed and evaluated across five LLMs and 740 safety-critical tasks under five defense settings, including real-world safety mechanisms.
Abstract
LLM agents that invoke external tools face critical safety vulnerabilities when malicious manipulations exploit their implicit trust in tool outputs and metadata. However, identifying these vulnerabilities through testing is challenging due to the need to bypass safety guardrails with semantically legitimate inputs, the difficulty in characterizing successful attacks amid implicit tool trust, and the requirement to maintain logical consistency across fragile state-dependent execution chains. In this paper, we first conduct an empirical study to investigate how external tools influence agent reasoning. Guided by the findings, we propose Datura, an automated red teaming testing framework that exposes safety vulnerabilities through chained tool manipulation. Through a five-stage workflow, Datura dynamically generates test cases where each individual step appears legitimate yet collectively leads to harmful outcomes. We evaluate Datura across five LLMs and 740 safety-critical tasks under five defense settings, including real-world safety mechanisms. Under Model Alignment, Datura achieves 94.86--99.59% attack success rate (ASR), outperforming the strongest baseline by up to 25.27 percentage points. Under Prompt Refuge, Datura maintains 78.78--95.54% ASR, showing that progressive tool-chain manipulation remains effective even under prompt-level safeguards.
ToFu is presented, a tool-aware framework that automatically derives semantic security checks for LLM tools from their existing metadata and a CWE-guided sanitizer corpus and automatically generates fuzzing harnesses and semantic oracles using runtime introspection and outcome sanitization.
Md. Nazmul Islam Ananto, Pierre Prevost-Avenel, Colin Smith et al.· 0 citations
It is suggested that typed tool mediation and host-side world-effect evaluation can improve LLM-agent execution safety while preserving benign-task utility.
Jun-Hao Zhou, Tao Jia· Proceedings of the 41st IEEE...· 0 citations
A tool-using model can follow a malicious instruction even when its credentials are valid. We study whether task-scoped authorization contains the resulting tool execution. Our paired-replay testbed samples a model request once and submits the same action, resource, and arguments to broad bearer, scoped JWT, sender-con...
This work introduces ToolFence, which compiles a typed authorization blueprint before execution, enforces it through a deterministic monitor, and when the blueprint is incomplete asks a judge to grant new capabilities rather than adjudicate each concrete call, improving runtime efficiency.
Yan-Jie Li, Xiang-Yu He, Xue-Long Dai et al.· arXiv.org· 3 citations
VEX-Bench is introduced, the first benchmark for evaluating LLM agents'ability to assess the exploitability of software supply chain vulnerabilities, and contains 75 real-world cases mined from GitHub and labeled by security experts, covering Python, Java, and Go.
Jia-Hao Shi, Edward Tsien, Yi-Feng Di et al.· 1 citation
CIPR (Coding In Poisoned Repos), the first benchmark that systematically varies PLCs in poisoned real-world repositories, is introduced and highlights that coding agent vulnerability is not a static property, but a dynamic outcome shaped by everyday user configurations.
Fu-Kang Zhu, Bin-Bin Zhao, Rui-Xiao Lin et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.