LLMKernelBench: Benchmarking Large Language Models on Software Vulnerability Detection in Linux Kernel

Abstract

Large language models (LLMs) demonstrate strong capabilities in code-related tasks, however their effectiveness in software vulnerability detection (SVD) remains poorly understood due to inadequate evaluation frameworks. Existing benchmarks suffer from training data contamination, isolated function evaluation without cross-component context, lack of strict vulnerable-patched pairing, and binary classification without hierarchy-aware common weakness enumeration (CWE) assessment, which prevents a reliable measurement of security reasoning versus pattern matching on leaked data. We introduce LLMKernelBench, a rigorous benchmark comprising 417 real-world Linux kernel vulnerabilities across 74 CWE types, split into a primary benchmark dataset (PBD; 314 samples, $\leq$2024) and a leakage free dataset (LFD; 103 samples, 2025 post-cutoff), with context-aware evaluation at three granularity levels and hierarchy-aware metrics quantifying semantic proximity in misclassifications. We evaluate seven LLM spanning code-specialized and general-purpose architectures. Binary vulnerability detection is near-random ($\sim\!\! 50\%$ accuracy) and strongly biased: some models label $>75\%$ of samples as vulnerable, while others mostly label them as nonvulnerable. CWE prediction is effectively unusable, with an average Top-1 accuracy of 1.4% (best: 3.3%) and a 12.3% hierarchy proximity score, providing little reliable exact or taxonomy-level signal. On the leakage-free 2025 split, binary accuracy remains near-random and robustness to multifile abstraction is model-specific rather than tied to specialization, with code-specialized and general-purpose models degrading by 2.6% and 4.5% on average, respectively. The micro-to-macro CWE-accuracy gap is larger on LFD (6.6 points) than on PBD (0.9 points), which is consistent with sensitivity to class frequency but does not identify an internal model mechanism.