PonyEval: Evaluating LLM-Based Program Repair for Capability-Safe and Actor-Oriented Pony Software

Abstract

Repository-level issue-resolution benchmarks have made executable evaluation central to software-engineering agents, but their language coverage remains concentrated in mainstream ecosystems. Pony presents a different regime: it combines actors, reference capabilities, ahead-of-time compilation, and a rapidly evolving historical toolchain, making both patch generation and faithful replay difficult. We introduce PonyEval, a SWE-bench-style benchmark of 291 real GitHub issue-pull-request pairs from 15 Pony repositories. Every instance binds an issue statement, a historical base commit, a developer gold patch, a black-box test patch, and a reproducible runtime mapping. The frozen release passes an offline audit requiring the issue-specific test to fail on the base state and pass after the gold patch; it contains no duplicate instance identifiers or canonical repository-PR pairs. In a separate full-set semantic selection audit, three isolated machine reviewers label all 291 instances as include or exclude; their Fleiss'kappa is 0.8968, with 249 unanimous inclusions and 32 unanimous exclusions. To replay eleven years of repository history, we reconstruct 72 runtime images covering 289 unique base commits and verify their availability on five heterogeneous compute nodes. We define a matched evaluation with mini-SWE-agent 2.4.6 for GPT-5.6-sol, DeepSeek-V4-Pro, GLM-5.2, MiniMax-M3, and Kimi-K3, followed by strict patch application, compilation, and hidden-test validation. Across the patches actually produced by each model, conditional resolution rates range from 10.21% to 24.68%. These rates characterize the quality of generated patches rather than success over all 291 benchmark tasks.