Following retrieved evidence does not guarantee factual correctness: misleading evidence can induce a model to replace an answer it previously gave correctly. Standard accuracy measures obscure this behavior by combining answer replacement with preexisting errors. We introduce RAG-Stress, a controlled diagnostic protoc...
Shun-Yuan Zhou, Hao Chen, Tian-Yu Wang et al.· 1 citation
Static-analysis checker synthesis requires agents to interpret a defect specification, inspect a repository, implement analyzer-specific logic, and refine the checker through repeated compilation and analysis feedback. Existing coding-agent benchmarks focus on tasks such as patch generation or vulnerability detection a...
Hang He, Li-Sheng Wang, Hao Chen et al.· 3 citations
LLM agents use skills to improve performance on specialized tasks. To complete a user request, an agent may invoke several skills in sequence, allowing information produced under one skill to guide the next. Because skills may come from open-source repositories, this handoff can also carry attacker-controlled claims in...
Tian Dong, Zi-Xuan Ma, Haodong Zhao et al.· 1 citation
This work shows that observation contract compliance is an emergent, regression-prone capability: it is neither guaranteed by general tool-use ability nor consistently improved by larger or newer models.
Ji-Cheng Wang, Yifeng He, Zi-Li Wang et al.· arXiv.org· 1 citation
Findings show that task utility does not imply privacy or authorization compliance and that authorization in community settings remains an unresolved security challenge for persistent LLM agents.
Hao Chen, Wen-Hui Dong, Ye Chen et al.· 0 citations
VulnGym is a real-world repository-level benchmark for evaluating vulnerability detection by coding agents that aligns reviewed GitHub advisories with their corresponding vulnerable version repositories and defines an end-to-end detection task and three oracle-based subtasks to jointly evaluate vulnerability detection...
Kexing Ji, Jiachen Liu, En-Ze Hu et al.· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.