Skip to content

ReproBench: Benchmarking LLM Agents on Reproducing Vulnerability From Scratch

Sep 2026 · arXiv.org · Vol abs/2609.34450 · 1 citation · 38 references
Computer Science

TL;DR

ReproBench is presented, an evidence-grounded benchmark designed to evaluate agent capabilities in end-to-end vulnerability reproduction starting from solely a CVE identifier, and confirms that state-of-the-art LLM agents already possess the capacity for fully autonomous end-to-end vulnerability reproduction.

Abstract

Large language model (LLM) agents are increasingly evaluated on cybersecurity tasks such as vulnerability reproduction, exploitation, and patching. However, existing cybersecurity benchmarks predominantly operate under a post-environment evaluation paradigm, i.e., handing the agent source code, a container, or an executable binary. This setup bypasses the critical environment reconstruction step, leaving a fundamental question for real-world vulnerability analysis: can an agent autonomously reconstruct the required execution environment and reproduce a vulnerability entirely from scratch? To address this gap, we present ReproBench, an evidence-grounded benchmark designed to evaluate agent capabilities in end-to-end vulnerability reproduction starting from solely a CVE identifier. ReproBench decomposes the full reproduction workflow into six distinct phases, and assesses performance on each phase independently using verifiable experimental artifacts: downloaded firmware images, unpacked binaries, granular analysis logs, and validated crash samples, among others. We instantiate ReproBench with 30 real-world IoT firmware vulnerabilities, which serve as ideal test cases for our from-scratch evaluation setting. Our evaluation demonstrates that 45.3% of test runs resort to vulnerability simulation - a prevalent remediation workaround adopted across all evaluated LLM agents - while only 5.3% of CVE-model pairs achieve successful reproduction of real-world vulnerabilities. Despite the low overall success rate, these non-trivial successful cases confirm that state-of-the-art LLM agents already possess the capacity for fully autonomous end-to-end vulnerability reproduction. Concurrently, our in-depth analysis of failed cases identifies core bottlenecks impeding LLM agents throughout the reproduction pipeline, offering actionable insights for subsequent research.

View source

Similar papers

#artificial intelligence Preprint Oct 2026

EvoRiskBench: An Evolving Benchmark for Runtime Security Risks in Workspace Agents

Workspace agents combine large language models with execution harnesses to perform stateful, multi-step tasks that access or modify external resources. Existing benchmarks leave gaps in executable coverage of their runtime security risks, while evolving model capabilities, harnesses, tools, and threats motivate benchma...

Shi-Yi Kuang, Xue-Mei Luo, Kun Liu et al. · 0 citations

Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale

VLoc Benchmark results establish vulnerability localization as a distinct repository-scale capability and provide a setting for studying both how security agents search for vulnerable code and when they should refrain from reporting it.

Aman Priyanshu, Supriti Vijay, Kimia Majd et al. · 2 citations · ⚡1
Preprint Oct 2026

SecLog-Bench: A Scalable Benchmark for Cybersecurity Log Reasoning with Scaffolded Evidence

This work builds an extensible simulation platform that covers 29 attack scenarios, supports repeated execution and the integration of new scenarios, and enables scalable generation of cybersecurity logs with attack-stage labels and annotates each sample with the essential context for cybersecurity log reasoning and a...

Yi-Fan Wang, Dong-Liang Xu, Xiao-Yong Li et al. · 0 citations
#artificial intelligence Preprint Sep 2026

SWE-Test: Benchmarking LLM Vulnerability Discovery via Input Prediction

This work recasts vulnerability discovery as an input-prediction task with a closed, deterministic ground truth, and decomposes discovery into three task modes over 22 real-world C/C++ programs spanning 15 domains, finding constraint inference, not navigation, is the dominant bottleneck.

Yuan-Xiang Shi, Jia-Yi Lin, Xuan-Yong Lin et al. · 0 citations
Preprint Oct 2026

Discovery or Memorization? A Faithful Benchmark for Real-World Business-Logic Vulnerability Discovery

The rapid advancement of cybersecurity agents in vulnerability discovery has increased the need for benchmarks that evaluate their capabilities accurately. However, synthetic benchmarks lack realism, real-life benchmarks are prone to contamination, and many existing benchmarks have already been saturated by current mod...

Rui Yang, Yi-Chao Xu, Zhuo-Hao Zhang et al. · 0 citations
#machine learning Preprint Sep 2026

VEX-Bench: Benchmarking LLM Agents for Assessing Exploitability of Software Supply Chain Vulnerabilities

VEX-Bench is introduced, the first benchmark for evaluating LLM agents'ability to assess the exploitability of software supply chain vulnerabilities, and contains 75 real-world cases mined from GitHub and labeled by security experts, covering Python, Java, and Go.

Jia-Hao Shi, Edward Tsien, Yi-Feng Di et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.