PREreview of "Towards a Science of AI Agent Reliability"

Abstract

This Zenodo record is a permanently preserved version of a PREreview. You can view the complete PREreview at https://prereview.org/reviews/23179310. ## Summary This paper argues that compressing agent behavior into a single success metric obscures the operational flaws that matter in deployment. Grounded in safety-critical engineering, the authors propose twelve concrete metrics decomposing reliability along four dimensions: consistency, robustness, predictability, and safety. They evaluate 15 models across two benchmarks and find that recent capability gains have yielded only small improvements in reliability. An interactive dashboard accompanies the paper. ## Strengths The central claim is correct and long overdue: a 70% benchmark score tells you almost nothing about what the 30% of failures look like. Grounding the framework in safety-critical engineering is the right lineage. The four dimensions are well chosen; consistency across runs is the most neglected metric and maps directly onto what deployment gates should assert. The headline finding — capability gains translating into only small reliability gains — is important and should change how teams interpret leaderboard progress. ## Major concerns and questions 1. Twelve metrics need prioritization. Which two or three actually discriminate between models in practice? A practitioner with a finite evaluation budget needs to know where to start. 2. The cost of measuring reliability is unaddressed. Consistency metrics require many runs; for long-horizon agents each run costs real money. Reporting the compute cost of the full profile — and a "budget" subset — would be valuable. 3. The findings rest on two unnamed benchmarks. Do they resemble real deployment tasks, or the academic evaluations the paper critiques? 4. Predictability and safety need crisper operational definitions tied to measurable quantities. 5. Positioning relative to existing repeated-trial metrics (pass^k) would clarify the marginal contribution. ## Minor points - A permanent archived link for the dashboard would guard against link rot. - A short "how to adopt this in your eval harness" checklist would bridge framework and practice. ## Overall assessment An important paper that formalizes what production experience already suggests: we are measuring the wrong thing. Prioritizing the metrics and costing the measurement would turn a strong research contribution into an operational one. Competing interests The author declares that they have no competing interests. Use of Artificial Intelligence (AI) The author declares that they used generative AI to come up with new ideas for their review.