Automated building design must comply with seismic and wind codes and satisfy structural mechanics constraints, yet most existing agents produce visually plausible models without verification grounded in mechanical analysis and code compliance. We introduce the Vibe Building task and propose PE-Loop (Physics-Engine-in-...
Yong-Qing Jiang, Hao-Ran Luo, Jian-Ze Wang et al.· 0 citations
Modern coding agents can deliver increasingly large repository-level changes, and recent benchmarks reflect this by emphasizing long-horizon tasks with large reference implementations. Many benchmarks evaluate coding agents'implementation capability to produce correct code edits from detailed specifications. However, p...
Yun Peng, Zi-Han Wu, Ze-Yang Zhuang et al.· 0 citations
Vul4Py is presented, a Python AVR benchmark in which every entry carries a paired oracle: an exploit oracle that must fail on the vulnerable revision and pass on the fixed one, together with a project-native pytest functional oracle that must pass on both.
Bui Tan, Ting Zhang, Ferdian Thung et al.· Lecture notes in computer sc...· 2 citations
TraceProbe is presented, a trajectory-diagnostic framework that recovers what resolve rate hides and adds auditable diagnostic context to outcomes by localizing inspection targets, suggesting failure hypotheses, and prioritizing runs for review.
Rui Shu, C. Chong, Xin Zhou et al.· arXiv.org· 1 citation
SWE-RPG is introduced, a repository-level benchmark that combines executable patch evaluation with validated ground-truth references (GTs) for Requirement Clarification and Implementation Planning, and suggests implicit-requirement recovery as a key candidate direction for improving coding agents.
Xin Zhou, Chun-Yong Chong, Kisub Kim et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.