Codoku: Renewable Program-Reasoning Challenges for Frontier Coding Agents
Existing program-reasoning benchmarks ask large language models to predict a program's behavior on a given input. Coding agents break two assumptions on which these benchmarks rest: an agent can recover the answer by executing the program instead of reasoning about it, and fixed task sets drawn from existing programs a...
Cong Li, Hao Sun, Ze-Nan Li et al.
· 0 citations