It is indicated that current coding agents cannot yet reliably reason about execution in complex systems codebases, and that paired evaluation is necessary to measure this ability.
Chen-Xi Huang, Hao-Nan Wang, Jing-He Zhang et al.· 0 citations
LLM agents are increasingly used to automate vulnerability detection, but reliable detection is challenging because agents must distinguish subtle, security-relevant fixes from benign code variation. While underlying LLM capability has improved significantly, the accuracy of the agent still depends heavily on the harne...
Ira Ceka, Arnav Garg, Ryan Lutz et al.· 0 citations
Coding agents are typically provided end-to-end behavioral specifications that the code they produce should satisfy, such as tests and input--output examples. How many such specifications must a developer provide, and which ones, for the agent to produce the developer's intended program? We argue that this cannot be de...
Aditya Golatkar, Alessandro Achille, Renos Zabounidis et al.· 0 citations
This study traces the full decision-making pipelines of the 5 APR agents across 500 real-world repair tasks, from issue description to patch validation, and reveals that, while agents excel at simple fixes, they struggle with logic-intensive bugs.
Ira Ceka, Saurabh Pujar, Shyam Ramji et al.· Proceedings of the ACM on so...· 0 citations· ⚡1
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.