#natural language process...
Jul 2026
ORCA-bench: How Ready Are Language Model Agents for Oncall?
The ORCA-bench benchmark is introduced, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting and is a lower bound on the engineering investment required before frontier coding agents can be safely entrusted with production reliability.
Albert Gong, Kyuseong Choi, Abhineet Agarwal et al.
· arXiv.org · 1 citation