Skip to content
Open access

Datura: Progressive Red Teaming Testing for Tool Invocation Chain in LLM Agents

Oct 2026 · Proceedings of the ACM on Software Engineering · Vol 3, pp. 221 - 245 · 0 citations · 95 references

TL;DR

Datura, an automated red teaming testing framework that exposes safety vulnerabilities through chained tool manipulation, is proposed and evaluated across five LLMs and 740 safety-critical tasks under five defense settings, including real-world safety mechanisms.

Abstract

LLM agents that invoke external tools face critical safety vulnerabilities when malicious manipulations exploit their implicit trust in tool outputs and metadata. However, identifying these vulnerabilities through testing is challenging due to the need to bypass safety guardrails with semantically legitimate inputs, the difficulty in characterizing successful attacks amid implicit tool trust, and the requirement to maintain logical consistency across fragile state-dependent execution chains. In this paper, we first conduct an empirical study to investigate how external tools influence agent reasoning. Guided by the findings, we propose Datura, an automated red teaming testing framework that exposes safety vulnerabilities through chained tool manipulation. Through a five-stage workflow, Datura dynamically generates test cases where each individual step appears legitimate yet collectively leads to harmful outcomes. We evaluate Datura across five LLMs and 740 safety-critical tasks under five defense settings, including real-world safety mechanisms. Under Model Alignment, Datura achieves 94.86--99.59% attack success rate (ASR), outperforming the strongest baseline by up to 25.27 percentage points. Under Prompt Refuge, Datura maintains 78.78--95.54% ASR, showing that progressive tool-chain manipulation remains effective even under prompt-level safeguards.

Read PDF

Similar papers

Preprint Oct 2026

ToFu: Exploiting and Fuzzing the LLM-Orchestrator-Tool Trust Boundary

ToFu is presented, a tool-aware framework that automatically derives semantic security checks for LLM tools from their existing metadata and a CWE-guided sanitizer corpus and automatically generates fuzzing harnesses and semantic oracles using runtime introspection and outcome sanitization.

Md. Nazmul Islam Ananto, Pierre Prevost-Avenel, Colin Smith et al. · 0 citations
Preprint Oct 2026

Compromise Is Not Consequence: Evaluating Task-Scoped Authorization in LLM Agents with Paired Replay

A tool-using model can follow a malicious instruction even when its credentials are valid. We study whether task-scoped authorization contains the resulting tool execution. Our paired-replay testbed samples a model request once and submits the same action, resource, and arguments to broad bearer, scoped JWT, sender-con...

Tural Hagverdiyev · 1 citation

ToolFence: Fine-Grained Authorization for Secure Tool-Using LLM Agents

This work introduces ToolFence, which compiles a typed authorization blueprint before execution, enforces it through a deterministic monitor, and when the blueprint is incomplete asks a judge to grant new capabilities rather than adjudicate each concrete call, improving runtime efficiency.

Yan-Jie Li, Xiang-Yu He, Xue-Long Dai et al. · 3 citations
#machine learning Preprint Sep 2026

VEX-Bench: Benchmarking LLM Agents for Assessing Exploitability of Software Supply Chain Vulnerabilities

VEX-Bench is introduced, the first benchmark for evaluating LLM agents'ability to assess the exploitability of software supply chain vulnerabilities, and contains 75 real-world cases mined from GitHub and labeled by security experts, covering Python, Java, and Go.

Jia-Hao Shi, Edward Tsien, Yi-Feng Di et al. · 1 citation
#natural language process... Preprint Aug 2026

Beyond the Payload: How User Invocation Shapes Coding Agent Vulnerability to Repository Poisoning

CIPR (Coding In Poisoned Repos), the first benchmark that systematically varies PLCs in poisoned real-world repositories, is introduced and highlights that coding agent vulnerability is not a static property, but a dynamic outcome shaped by everyday user configurations.

Fu-Kang Zhu, Bin-Bin Zhao, Rui-Xiao Lin et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.