VICBench enables robust evaluation of vulnerability detection approaches and shows that state-of-the-art algorithms V-SZZ and LLM4SZZ achieve only 33.3%-40.1% F1, confirming that using existing approaches still entails significant manual effort.
Abstract
Evaluating security vulnerability detection tools requires benchmark datasets with vulnerability-inducing commits (VICs) - the commits that first introduce vulnerabilities into codebases. VICs are essential for determining the full range of vulnerable software versions. Existing vulnerability datasets suffer from limited programming language coverage, restricted patch complexity, and narrow project scope. Through our dual annotation by human experts and an agentic workflow, we create a benchmark - VICBench - of 100 verified VICs for 100 CVEs across 88 projects in Python, Java, and C++, covering 48 CWE types. VICBench features complex real-world vulnerability fixes averaging 38.6 lines and corresponding VICs of 252.5 lines - significantly larger than prior work. Our evaluation shows that state-of-the-art algorithms V-SZZ and LLM4SZZ achieve only 33.3%-40.1% F1, confirming that using existing approaches still entails significant manual effort. VICBench enables robust evaluation of vulnerability detection approaches.
This paper introduces SecVulEval, a context-aware benchmark designed to evaluate LLMs on vulnerability detection with rich contextual information, and believes it can serve as a foundation for advancing context-aware vulnerability detection with LLMs.
Md Basim Uddin Ahmed, Nima Shiri Harzevili, Jiho Shin et al.· AIware· 2 citations
VulnGym is a real-world repository-level benchmark for evaluating vulnerability detection by coding agents that aligns reviewed GitHub advisories with their corresponding vulnerable version repositories and defines an end-to-end detection task and three oracle-based subtasks to jointly evaluate vulnerability detection and diagnose limitations in code localization and evidence construction.
Kexing Ji, Jiachen Liu, En-Ze Hu et al.· 2 citations
Smart contract vulnerability detection requires evaluation protocols that separate real representation signal from dataset-specific artifacts. DIVE provides lifecycle-based tabular features for Ethereum smart contracts, but benchmark performance alone cannot show whether a dominant feature group is useful or only benefits from having many columns. This study examines Opcode Distribution features using 22,330 contracts, 397 processed features, and eight DASP-aligned vulnerability labels. Five multi-label learning configurations were evaluated under 3 x 5 repeated cross-validation, followed by global feature-group ablation, size-controlled random opcode ablation, per-label degradation analysis, cumulative stability analysis, and opcode-profile group-aware robustness checking. MultiOutput LightGBM achieved the best baseline performance, with Micro-F1 of 0.91396, Macro-F1 of 0.82464, and Macro-PR-AUC of 0.90146. Removing the full Opcode Distribution group reduced Macro-F1 to 0.78745, while removing a same-sized random opcode subset produced Macro-F1 of 0.82404. The findings indicate that Opcode Distribution acts as a collective predictive representation rather than a feature-count artifact, without implying causal vulnerability mechanisms.
Astrid Pranadani, Dhani Ariatmanto· Data Science Journal of Comp...· 0 citations
Model Context Protocol (MCP) servers have become essential for connecting large language models (LLMs) with external tools, data sources, and execution environments. However, their privileged access and bridging role introduce significant security risks. Despite many LLM-based detection methods, it remains unclear whether they can semantically differentiate paired vulnerable and fixed MCP servers rather than just identifying risky API calls. In this work, we present a dedicated benchmark for pairing MCP server vulnerability detection, comprising 75 paired vulnerable and fixed MCP tools synthesized by Gemini-1.5-Pro and manually cross-validated. We adopt the Matched Pair Accuracy (MPA) metric to evaluate detection methods. Claude Haiku-4.5 achieves the best MPA of 0.853, while GPT-5-mini attains 0.842 F1 but only 0.653 MPA, showing that conventional metrics can overestimate paired detection performance. Our benchmark provides a new perspective for systematic, reproducible, and MCP-specific security evaluation.
The Security Mutation Score (SMS) is proposed, a metric that classifies mutant kills into semantic, functional, incidental, and crash categories using operator-aware heuristics, distinguishing genuine security awareness from coincidental detection.
VIC-RAGENT is proposed, an LLM-based multi-agent framework for effective and explainable vulnerability detection that employs a multi-stage reasoning process that progressively refines candidate vulnerabilities through preliminary inspection, reanalysis, and a final decision stage.
Liyou Chen, Hailong Sun, Xiang Gao et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
