A production-trace validation protocol, optional LLM-judge integration, and maintenance of expectations as workflows evolve are described, a practical pattern for expressing behavioral contracts within an existing Java release pipeline.
Abstract
Infrastructure security does not establish whether an LLM agent selected an allowed tool, grounded its answer in tool output, or followed a required workflow. AgentTrustCI is a Java/JUnit 5 framework that expresses these application-layer expectations as policy, grounding, and trajectory checks over recorded traces. A deterministic replay implementation supports repeatable build-time tests; a judge interface provides an extension point for optional semantic evaluation. On 55 synthetic cases spanning seven enterprise domains, the full pipeline correctly classifies all 30 injected faults and 25 passing controls, including ten non-trivial controls, and misses none of 12 designated critical cases. Each single-layer configuration correctly classifies 35 cases because the corpus deliberately isolates ten faults per layer. A grounded answer obtained through a disallowed sequence illustrates the additional coverage of trajectory checks. Repeated measurements quantify validator execution separately from process startup and reporting. These results establish coverage of the authored regression suite, not production accuracy. We describe a production-trace validation protocol, optional LLM-judge integration, and maintenance of expectations as workflows evolve. The contribution is a practical pattern for expressing behavioral contracts within an existing Java release pipeline.
Results show that formal temporal monitoring adds value exactly when the trace exposes trustworthy history, and quantify how far current benchmarks are from that point and propose a twelve-field enforcement-ready trace schema.
N. Kekatos, Stylianos Basagiannis, Marinelio Chintri et al.· Proceedings of the 2026 8th...· 2 citations
Large language model (LLM) agents increasingly interact with tools and environments, yet plausible model decisions do not always translate into successful execution due to tool-call mismatches, interrupted workflows, and unsupported completion claims. We present EvoX, a Rust-based, multi-model agent runtime guided by T...
Haoyang Zhang, Qu-Meng Sun, Xiao Zhang et al.· 0 citations
AgentTriage is presented, an executable contract for replayable security diagnosis across heterogeneous runtimes whose claims cite canonical runtime events and requires abstention when mandatory evidence is missing and an open-source implementation and selected sanitized evidence are released.
Jie Ying, Xu-Tao Zhang, Ning-Yuan Chen et al.· 0 citations
Customized agent deployments define policies over tools, resources, and execution conditions. Providing these policies in the task agent's prompt does not reliably prevent violations. Training a guard requires examples of tool calls that comply with or violate those policies. We introduce AgentPolicyForge, a framework...
Chao Shang, Wan-Yu Du, Yi Zhang et al.· 0 citations
A tool-using model can follow a malicious instruction even when its credentials are valid. We study whether task-scoped authorization contains the resulting tool execution. Our paired-replay testbed samples a model request once and submits the same action, resource, and arguments to broad bearer, scoped JWT, sender-con...
An agent can complete a task while exhibiting undesirable behavior during execution. Developers need tests for the specific behaviors encountered in deployment, beyond fixed benchmark suites. We present TraceDance, an agent system that constructs targeted benchmarks from deployment traces for user-specified undesirable...