Static coding benchmarks are fixed at release. Over time their tasks leak into training data, and stronger agents eventually saturate them. We argue that a benchmark should instead aim to be self-scaling: its tasks and oracles are created at evaluation time, so no content exists in advance to leak, and its difficulty i...
Wei Minn, Zhensu Sun, L. K. Shar et al.· 0 citations
Vulnerability-detection benchmarks score the verdict an agent reaches, not the evidence it gathered. A model that recalls a CVE from pretraining therefore scores the same as one that traced the data flow. We study a task where this difference matters, deciding whether a commit introduces a vulnerability. Instead of sco...
Yi-Kun Li, Jin-Feng Jiang, Yuheng Yieh et al.· 0 citations
This work introduces Path2Spec, a divide-and-conquer framework that leverages LLMs to extract all execution paths from an input program, generates path-specific specifications for each, and merges them into a comprehensive overall specification.
Dan Huang, Zhensu Sun, Hui-Hui Huang et al.· 0 citations
Evaluations across agent systems, benchmarks, and backbone LLMs under 65 fault configurations show that all systems degrade under fault injection, suggesting that robustness depends on system implementation rather than model capability.
Gou Tan, Zhensu Sun, Jie-Ke Shi et al.· Proceedings of the 41st IEEE...· 6 citations
VulAgentRL is proposed, an agentic RL framework for interprocedural vulnerability detection built on a Code Property Graph (CPG), which outperforms state-of-the-art baselines, including frontier models, on the strict pair-wise-correct metric while issuing fewer tool calls, and its advantage persists on an out-of-distri...
Yi-Kun Li, Ting Zhang, Jia-Kun Liu et al.· arXiv.org· 2 citations
Experiments on real-world vulnerabilities show that CoSA consistently outperforms function-level and pure-LLM baselines, suggesting that explicit, metric-oriented repository context retrieval is crucial for practical and reliable automated severity assessment.
Jin-Feng Jiang, Yi-Kun Li, Cheng-Ran Yang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.