BDD-Bench: A Behavior-Driven Benchmark for Coding Agents across the Application Lifecycle

Abstract

The main contribution of this paper is BDD-Bench, a repository-level benchmark in which coding agents implement executable Gherkin specifications across an application's development lifecycle. Its 111 chronologically ordered instances come from five Python repositories. In lifecycle mode, the agent solves consecutive instances on an evolving codebase, carrying each solution forward. Original tests are augmented with visible and hidden model-generated scenarios to strengthen behavioral coverage and assess generalization. Across thirteen models evaluated under a single agent harness with three runs each, the best-performing model, Claude Opus 4.8, resolves 45.0% of instances, covering 29.5% of the production code in the benchmark's reference patches when weighted by size. We further introduce a fine-grained failure taxonomy that pinpoints why lifecycle chains stall, and a matched-rerun design that isolates how agent-authored versus reference code histories shape an agent's downstream progress. These results show that even the strongest configurations often pass visible tests but fail withheld variations of the same behavioral specification.