dspy-security-bench: reproducible security, authorization, and mission assurance evidence for tool-using AI agents

Abstract

A Python harness for measuring how well language-model agents resist indirect prompt injection. It wraps the AgentDojo task environments and adds a frozen, hashed measurement protocol, joint reporting of task utility alongside attack resistance, cluster-bootstrap confidence intervals over task pairs, and a confirmed/provisional criterion that decides when a result is stable enough to state as a claim. ImpactTwin adds controlled procurement pairs, functional side-effect evidence, repeated-execution uncertainty, and content-addressed community submissions. ProofRun adds a reusable trusted builder, GitHub/Sigstore provenance for exact evidence bytes, and an explicit evidence ladder. Native framework bridges connect OpenAI Agents SDK, LangChain, Pydantic AI, CrewAI, AutoGen, MCP, and custom loops to the same framework-neutral contract. Results are generated from committed evidence so rows and submissions can be audited offline. ControlTwin compares policy-off and policy-on functional outcomes, separates harm containment from safe mission recovery and clean utility, and binds the exact normalized policy to offline-verifiable evidence. RepeatControlTwin repeats the paired policy experiment with fresh agents and alternating condition order, then reports uncertainty bounds, functional transitions, recovery stability, clean-utility preservation, and separated condition-level usage. The Open Control Evidence Registry packages those policy-bound experiments for offline recomputation, public comparison, GitHub/Sigstore provenance, and independently reviewable contribution. IncidentTwin adds an inert cyber-response digital twin with functionally observed alert, secret, network, isolation, and critical-service outcomes. FederalProof binds verified repeated evidence to owner-supplied deployment context and exports OSCAL 1.2.2 assessment results, conditional POA&M inputs, an impact-assessment annex, a QASP scorecard, and a content-addressed manifest. MissionForge adds a strict data-only contract for agency- and company-owned mission evaluations. Its built-in SourceTwin protocol measures citation faithfulness, completeness, sufficiency, current-primary preference, clean utility, and injection resistance through structured claims and source IDs. AuthorityTwin adds a vendor-neutral delegated-authorization adapter contract, ten clean/adversarial identity and authority pairs, normalized request-bound decision receipts, simulated-effect containment, repeated uncertainty, content-addressed public evidence, ProofRun provenance, and FederalProof assessment export. InventoryForge turns bounded public AI-use-case inventories into contact-free, tamper-evident normalization reports and explicitly synthetic MissionPack drafts requiring accountable review. AgentGraphTwin traces six multi-agent authorization-path mutations, attributing first unsafe edge and synthetic blast radius. AuthorityBridge provides translation contracts for OPA, Cedar, OpenFGA, OAuth-bound MCP tools, and SPIFFE. ContinuousProof compares verified evidence identities and metrics using owner-supplied thresholds. AcquisitionProof exports vendor-neutral mission test plans, owner-defined QASP objective inputs, portability checks, cost-observation fields, and reevaluation triggers without automating a procurement decision. TraceProof converts operator-supplied OpenTelemetry JSON into privacy-bounded, pseudonymized evidence; applies deterministic authorization and external-effect rules; and exports synthetic replay twins, SARIF, and OSCAL observations. AgentGraphTwin v2 adds temporal ordering, token exchange, delegation continuity, step-up approval, revocation, parallel races, and multi-effect boundaries. ValueProof computes measured mission economics without forecasts or rankings. MissionPack Commons adds self-contained Ed25519 envelopes and a separately governed, content-addressed catalog for community mission protocols. The TraceProof Runtime Kit records metadata-only tool-boundary events across seven agent-framework profiles and tests sanitizer and MCP authorization evidence without retaining application content. ScheduleProof exhaustively explores bounded authorization-event interleavings, reports exact schedule coverage and minimal causal counterexamples, and exports offline-verifiable JSON and SARIF without executing a model or tool. CausalProof converts structural OpenTelemetry parentage into a provenance-separated ScheduleProof draft while keeping owner assertions, generic span links, and wall-clock candidates distinct. Native OpenAI Agents SDK and LangGraph bridges emit pseudonymized structural evidence and explicit atomic-event bindings without inspecting application content. CollectiveGuard analyzes content-free structural event records for autonomous agent collectives, detecting unapproved cross-run communication, indirect egress, peer-authority laundering, credential misuse, evaluator access, unsafe persistence, missed response windows, recovery approval failures, and non-independent or collapsed defenses with offline-verifiable JSON and SARIF.