This paper evaluated Niyam-AI on 2,000 real-world agent scenarios from Agent-SafetyBench and compared it against three existing safety approaches: NeMo Guardrails, Meta's Llama Prompt Guard 2, and OpenAI's GPT-OSS-Safeguard.
NiyamAI is presented, an intent bound runtime guardrail architecture providing cryptographically verifiable execution integrity for autonomous agents and subjects its own enforcement mechanism to 18 adversarial vectors across six classes, disclosing two implementation vulnerabilities identified and remediated during development.
Aditya Katkar, Om Karkele, Kartik Mandhane et al.· 1 citation
LLM agents are stateless and rely on external memory to carry context between steps. Because agents treat that memory as trustworthy, an adversary who can write to it can steer their behavior. The FARMA attack does this with no malicious command: it inserts fabricated entries into the agent's reasoning memory claiming a required safety step is already done, so the agent skips it. SENTINEL, the defense proposed with FARMA, scores entries against a fixed list of suspicious wordings; its authors note that an attacker who knows the list can reword the forgery and evade it, and leave this open. We show the gap is worse than stated. An automated attacker that simply asks a language model to reword the forgery evades SENTINEL on its first try, reducing its protection to zero on every model tested. We also find a capability paradox: the attack succeeds far more often on stronger models (98-100% on GPT-4o and GPT-4o-mini) than on Llama-3.1-8B (44%), because more capable agents follow reworded claims more faithfully, so the threat grows with capability. We propose Proof-of-Execution Memory (PoEM), which does not inspect memory at all. PoEM keeps a separate, tamper-evident, HMAC-chained ledger of the safety steps that actually executed, writable only by the trusted action layer, and allows a skip only if the ledger confirms real execution. An attacker can change what memory says but cannot forge a ledger entry for a step that never ran, so rewording no longer helps. Across three models and three scenarios, PoEM drives attack success to 0% while leaving legitimate operation intact (0% false positives in eight of nine cells, 1.7% in the ninth, within sampling noise), whereas SENTINEL wrongly blocks 33-50% of legitimate operations. PoEM also withstands attacks aimed at itself, adds microseconds of overhead, and works unchanged in a real LangChain agent. PoEM protects exactly the decisions it gates.
This work builds a working instance on a hierarchical multi-agent system, runs it under benign and attacked conditions across five language models and two task domains, and measures how much of that warning rests on removable surface cues of the attack rather than on its distributed structure.
D. Arias, Dev Prashant Mistry, Ren Wang et al.· arXiv.org· 0 citations
AI coding agents produce code faster than humans can review it. In our approach, the prover is the judge of whether the code is correct. Under a verifier-driven loop, AI agents wrote and verified bare-metal security software in Ada/SPARK spanning classical and post-quantum cryptography, TLS 1.3, IKEv2, X.509, and a Matrix client. GNATprove discharged 49,280 proof obligations, established functional correctness for selected primitives, and proved the absence of run-time errors for the rest, at roughly 20-40 times lower supervision cost than comparable hand verification. GNATprove alone was insufficient: some defects could not be detected and were resolved using known-answer tests, interoperability, or human review of specifications. Given weak checks, the agent tried to bypass them and reported success. We report where each layer caught faults and draw the central lesson: what an agent can be trusted to establish is bounded by the strength of its feedback.
The threat model in the HPC setting is defined, attack surfaces created by schedulers, shared storage, multi-project accounts, and scientific workflows are identified, and where current controls fall short is examined.
Software developers increasingly delegate routine programming work to autonomous AI coding agents that read untrusted project files, run commands, and call external services with limited human review. No developer can audit every action such an agent takes, so oversight defaults to a single artifact: the summary the agent writes when it finishes. The completeness of that summary is a security property in its own right, yet current evaluations do not measure it. Existing benchmarks ask whether a malicious instruction planted in a repository succeeds in redirecting an agent. They do not ask whether the agent then discloses what it did.
This study takes up the second question. We instrument an open-source coding agent running against a controlled, sandboxed repository carrying an injected instruction, and establish per-trial ground truth on what the agent actually did from channels independent of its own account. The design covers nine models, with a second agent used to check that the effect reproduces. Setting that ground truth against the report the developer receives separates two behaviours that prior work has treated as one: whether an attack succeeds, and whether the agent discloses it.
In our data, resisting attack more often does not make a model more forthcoming when an attack does succeed. The two vary independently, so a benchmark scored on attack success alone cannot rank agents on the risk a developer actually carries. We find preliminary evidence that a single sentence added to the developer's own prompt substantially improves disclosure.Faculty Supervisor: Rongxing Lu
