AI Agents Do Not Fail Alone:The Context Fails First

Fouad Bousetouane

Jul 2026 · arXiv.org · Vol abs/2607.14275 · 1 citation · 24 references

Computer Science

TL;DR

These findings establish context measurement as a validated preflight signal for agent reliability and position context engineering as an auditable layer of agent evaluation and governance.

Abstract

Context engineering has become central to building reliable AI agents, yet it remains largely unmeasured. Agents do not fail in isolation: their behavior is shaped by the instructions, tools, memory, retrieved knowledge, guardrails, and untrusted inputs accumulated in their context. When this context is weak, agents drift, hallucinate, misuse tools, ignore constraints, become vulnerable to injection, and waste tokens. This paper validates context-engineering quality as an independent leading indicator of agent reliability. We implement the measurement in ProofAgent-Harness, an open-source infrastructure for AI agent evaluation that uses multi-juror, consensus-based scoring. The harness assesses context across seven criteria: role clarity, guardrail coverage, instruction consistency, tool schema quality, grounding sufficiency, injection hardening, and token efficiency. Crucially, the context score is isolated from behavioral metrics and release decisions, enabling a non-circular validation. Through a controlled context-quality study across regulated agent domains, holding frontier LLM agents fixed and varying only their operating context, we show that context-quality criteria consistently predict their corresponding behavioral outcomes. Grounding sufficiency predicts hallucination resistance, guardrail coverage predicts manipulation resistance, instruction consistency predicts instruction following, and tool-schema quality predicts tool use. These findings establish context measurement as a validated preflight signal for agent reliability and position context engineering as an auditable layer of agent evaluation and governance.

DOI arXiv Semantic Scholar

View source

Similar papers

Jul 2026

Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories

It is shown that borderline task difficulty is agent-specific (Spearman rho=0.75), offering a candidate explanation for prior contradictions: single-agent studies draw tasks from different agents'informative bands.

Prakhar Khatri · arXiv.org · 2 citations

Review Aug 2026

Agent Safety Should Be a Runtime Contract

The right unit of safety in agentic AI is the trajectory-with-checkable-evidence, not the model, and this work formalizes an Agent Trajectory Schema and Evidence Chain, state a compositional gating proposition based on standard monitor composition, and outline a research agenda.

Albus W. Ng, Yibin Han, Jusheng Zhang et al. · 0 citations

Jul 2026

AgentAbstain: Do LLM Agents Know When Not to Act?

This work presents the first systematic evaluation framework for agentic abstention, and identifies failure modes such as post-hoc abstention, in which agents execute irreversible actions before recognizing abstention triggers.

Xun Liu, Y. Zhang, Vira Kasprova et al. · arXiv.org · 2 citations

Jul 2026

Do Agent Benchmarks Measure Capability? Protocol Validity in the Age of Agentic AI

Agent benchmarks increasingly evaluate repository editing, web research, terminal use, and long-horizon interaction. Their scores support capability claims only when the evaluation protocol keeps the intended capability necessary for success. Recent reward-hacking benchmarks and system reports show that agents can instead recover public solutions, read evaluation artifacts, infer generator structure, manipulate feedback, or benefit from invalid scoring paths; existing responses do not provide a common procedure for attributing these shortcuts and quantifying their effect across benchmarks. We formulate protocol validity and introduce HackDetect, a post-hoc audit that identifies an exposure, determines how the agent used it, and assesses whether the resulting score is misleading. We quantify score inflation with the Mislead gap, defined as the exploit score minus the intended score. We audit 2,385 traces across 15 agent benchmarks and find evidence of exposures and reward hacking in 67.0% of Frontier Science traces and 66.7% of AutoLab tasks. Across paired comparisons, we measure score inflation of 0.45-1.00, showing that benchmark reports should provide evidence that scores reflect the intended capability.

Jia-Qi Shao, Hanck Chen, Wei Zhang et al. · arXiv.org · 4 citations

Jul 2026

LOGOS: A Living Logic for AI Agent Teams That Evolve With Humans

This work introduces logos, a pluggable layer for self-evolution and governance that strengthens existing multiagent frameworks rather than replacing them, and provides a living logic for accountable automation.

Yuma Ichikawa, Yamato Arai, Kosaku Kimura et al. · arXiv.org · 0 citations

#artificial intelligence Preprint Aug 2026

Safety Does Not Compose: Non-Decaying Loop State for Autonomous LLM Agents

LoopHarness is presented, which restores a persistent, non-decaying safety state at the loop level at the loop level, and gives a complete evaluation protocol on native Agent-SafetyBench tasks with paired clean and attacked episodes, an outer-state attack suite whose decisive evidence exists only across iterations, per-module ablations, and an adaptive white-box red team.

Chenmin Wu, H. Jia, Yang Liu et al. · 0 citations