CARE-Bench: A Capability-Stratified, Cost-Aware Benchmark for Agentic Software-Security Detection

Abstract

Agentic systems for software-security detection have advanced quickly, from passive classifiers to cyber reasoning systems that autonomously find and patch vulnerabilities. Yet progress is hard to measure: systems are reported on incomparable datasets, with metrics that ignore cost, and on corpora whose labels are known to be noisy and prone to train-test contamination. We present CARE-Bench, an evaluation harness and protocol that makes results across heterogeneous agentic detectors directly comparable. CARE-Bench makes three design commitments. First, it is capability-stratified: every system is scored along the four levels of detection, localization, triage, and remediation, so that a single-call classifier and a multi-agent patcher are placed on the same ladder. Second, it treats cost as a first-class metric, reporting tokens, dollars, and wall-clock time, and dollars per true positive, so that accuracy and expense are reported together. Third, it is contamination-resistant, providing temporal-holdout and de-duplication controls that address the documented label-noise and leakage problems of existing corpora. CARE-Bench does not introduce a new vulnerability dataset; instead it unifies existing public corpora under one canonical schema through dataset adapters, and ships as open-source code with a reproducible runner, a metric suite, detector interfaces, and a leaderboard schema that rejects submissions omitting cost or contamination provenance. We describe the design, position it against recent benchmarks, and release the harness to support reproducible, cost-aware evaluation of the next generation of security agents.