Disclosure Failure in Compromised AI Coding Agents

Abstract

Software developers increasingly delegate routine programming work to autonomous AI coding agents that read untrusted project files, run commands, and call external services with limited human review. No developer can audit every action such an agent takes, so oversight defaults to a single artifact: the summary the agent writes when it finishes. The completeness of that summary is a security property in its own right, yet current evaluations do not measure it. Existing benchmarks ask whether a malicious instruction planted in a repository succeeds in redirecting an agent. They do not ask whether the agent then discloses what it did. This study takes up the second question. We instrument an open-source coding agent running against a controlled, sandboxed repository carrying an injected instruction, and establish per-trial ground truth on what the agent actually did from channels independent of its own account. The design covers nine models, with a second agent used to check that the effect reproduces. Setting that ground truth against the report the developer receives separates two behaviours that prior work has treated as one: whether an attack succeeds, and whether the agent discloses it. In our data, resisting attack more often does not make a model more forthcoming when an attack does succeed. The two vary independently, so a benchmark scored on attack success alone cannot rank agents on the risk a developer actually carries. We find preliminary evidence that a single sentence added to the developer's own prompt substantially improves disclosure.Faculty Supervisor: Rongxing Lu