An empirical study of whether complete software artifacts generated by LLM coding agents can be executed in a clean environment using only the code, dependency specifications, and instructions the agent provides suggests that coding-agent evaluation should treat clean-environment executability as a first-class metric alongside functional correctness.
These findings suggest that while AI agents excel at rigorous boundary testing, they lack the “environmental awareness” needed to write stable, hermetic tests.
Preet Jhanglani, Zeel Desai, Vidhi Kansara et al.· Annual International Compute...· 0 citations
End-to-end GUI testing is essential for verifying web applications, yet existing evaluations rely on predefined checklists and are confined to the data and frameworks of web generation benchmarks, leaving the bug-discovery ability of vision-language models (VLMs) systematically untested. We introduce \textbf{C}ode-driven \textbf{A}gentic \textbf{T}esting (CAT), a paradigm in which the agent writes Playwright code to drive the browser, gathers feedback, and autonomously explores web applications to uncover bugs. We instantiate CAT with CATJudge, an agentic framework that unifies Browser-Use and Computer-Use tools within a single environment and CATTest, a benchmark of 102 AI-generated web applications with carefully annotated bugs, built through close human-AI collaboration to feature complex interactions and subtle defects. Experiments with mainstream VLMs show that all evaluated models perform poorly, revealing a clear gap between current VLM capabilities and the demands of real-world testing in AI web development. We release our code and data at https://github.com/SleepyWithoutCoffee/CATJudge.
Bin Hong, Zhen-Chao Zhang, Ji-Yuan He et al.· 0 citations
Vero is introduced, the first benchmark to evaluate joint implementation and proof synthesis at the repository level and an audit mechanism where agents are allowed to formally prove unsatisfiability of provided specification or incorrectness of reference code, which surfaces and corrects latent code and specification errors during curation.
Zhe Ye, Hantao Lou, Yuechun Sun et al.· 0 citations
The AgentCodeReview system is presented, a multi-agent system that is able to conduct explainable code review and automated bug repair by leveraging software engineering agents with different code review tasks and its utility and extensibility to the field of explainable AI in software quality assurance are demonstrated.
B. N, T. L. Manasa· International journal of com...· 0 citations
This paper reviews recent empirical literature to ask what the developer's job is shifting from typing code to directing agents that type code a change often summarized as a move from code generation to code orchestration.
P. N. Nesarajan, P. Thenmozhi, Shenbaga Priya et al.· International Journal of Inn...· 0 citations
It is observed that generated code often omits basic input validation or memory-safety checks, which can lead to overflows, resource exhaustion, or other reliability/security issues, and even the largest models frequently make simple mistakes.
Rodrigo Pato Nogueira, Marco Vieira, João R. Campos· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
