Agentic Flakiness: Reframing LLM Agent Reliability Through the Lens of Flaky-Test Research
Abstract
Large language model (LLM) agents are nondeterministic: the same task, run twice under identical conditions, may succeed once and fail once. The agent-evaluation literature has begun to measure this- most directly through the pass^k metric introduced with τ-bench- but treats it as a novel phenomenon. It is not. Software engineering has studied the same phenomenon under the name flaky tests since at least 2014, and has produced a root-cause taxonomy, detection algorithms, cost models, and CI-level management policy. This paper argues that agent evaluation should import that body of work. We give a structural mapping from established flaky-test root causes to sources of agentic nondeterminism, showing the correspondence is mechanistic rather than metaphorical: batch-noninvariant GPU kernels are the agentic form of floating-point reduction-order flakiness. Building on the latent-success representation of recent Markov-chain reliability work, we prove that pass^k converges as k → ∞ to the agent's deterministic pass rate D, and is flat in k if and only if no task is flaky. This yields the Flake Gap Φ = pass^1? D: the portion of a benchmark score an agent cannot reproduce on demand. Finally we show Φ is not identifiable from currently published numbers-fitting a zero-and-one- inflated Beta latent model to reported τ-bench results constrains D only to [0.00, 0.42] on τ-retail, leaving the deployable pass rate uncertain by 61% of the headline score. Benchmarks should therefore publish per-task trial counts rather than aggregate pass^k
