Skip to content

PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents

Oct 2026 · 1 citation · 37 references
Computer Science

TL;DR

This work proposes PROBE (Policy Rule Observation via Behavioral Exploration), an algorithm for training LLM-based auditors that interact with deployed agents as realistic users and probe them for policy violations through multi-turn, tool-mediated dialogue, and formalizes agent auditing as a two-player Markov game.

Abstract

Modern agentic systems extend language models with external tools, enabling them to complete complex tasks through multi-turn interactions. As these agents are deployed in consequential settings, they are often constrained by policies governing pricing, privacy, confirmation, safety, and other requirements. However, existing evaluation methods leave a critical gap: capability benchmarks measure what agents can accomplish, while content-safety red-teaming measures harmful generations, but neither tests whether agents comply with predefined policies during otherwise valid tool use. Such policy violations are difficult to audit because their effects may emerge only after several turns, span multiple tool calls, or surface in downstream systems. We propose PROBE (Policy Rule Observation via Behavioral Exploration), an algorithm for training LLM-based auditors that interact with deployed agents as realistic users and probe them for policy violations through multi-turn, tool-mediated dialogue. We formalize agent auditing as a two-player Markov game and propose a composite reward that balances violation discovery, behavioral plausibility, conversation progress, and policy diversity. To study training objectives and reward design, we compare GRPO, DPO, and GFlowNet-based optimization in a controlled setting. Across extensive experiments, auditors trained with PROBE uncover substantially broader and higher-yield policy violations than same-architecture zero-shot adversaries, even when the auditor is significantly smaller than the target agent. Our results show that multi-turn behavioral auditing is a distinct and necessary axis of agent evaluation, complementing capability and safety benchmarks. PROBE offers a scalable path toward automated policy auditing for real-world agentic systems and suggests a co-evolutionary loop in which auditors expose failures and agents improve from auditor feedback.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

Think Before You Act: Decision Gate for LLM Agents

LLM agents increasingly perform consequential actions through external tools, yet their proposed actions are often executed without independent verification of policy compliance or task completion. We introduce JevAct, a runtime verification framework that intercepts selected agent actions before execution and enables...

Iftach Shoham, Yuval David, Offer Akrabi et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Policy-Verified Self-Training: Learning Agent Compliance from Verifiable Policy Rewards

Runtime policy verifiers can prevent agents from executing disallowed actions, but their decisions are typically used only for enforcement and discarded afterward. We ask whether these decisions can instead become verifiable policy rewards for training the agents they guard. We introduce Policy-Verified Self-Training (...

Minhyeong Yu · 0 citations

MATE: Policy-Aware Security Auditing for Mobile Agents via Synthesis-Driven Trajectory Learning

Mobile agents powered by foundation models now automate complex, multi-step workflows on real devices, but their trajectories can violate app-specific security policies. Existing trajectory-level defenses rely on LLM prompting or rigid rules, and thus fail to support fine-grained, natural-language policies that general...

Chang-Yue Jiang, Jia-Yi Wang, Xin Wen et al. · 2 citations

AgentBoundary: Counterfactual Evaluation of Safety in Tool-Using LLM Agents

This work introduces AgentBound, the first four-way counterfactual generation-and-evaluation framework for tool-using agent safety, and shows that effective agentic alignment requires action decisions to track permission-relevant execution evidence, rather than refusal strength alone.

Tian-Zhuo Yang, Zi-Rui Mi, Yan-Tao Huang et al. · 1 citation
#artificial intelligence Review Sep 2026

CART: Closed-Loop Adaptive Red Teaming for Large Language Models

CART turns red teaming from a one-time checklist into a continuous, adaptive, and auditable search for model and agent weaknesses, suggesting that contextual adaptation can reveal weaknesses that direct prompt replay does not exercise.

Dong-Dong Zhang, Teng-Chao Lv, Yi-Ling Jia et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.