Jul 2026· International Conference on Artificial Intelligence Testing· pp. 136-143· 0 citations· 23 references
Abstract
Automated vulnerability repair with large language models is increasingly woven into security workflows, yet evaluation practice has focused almost entirely on a single question: does the patch eliminate the targeted weakness? However, an equally important question has received little systematic attention: has the patching act itself introduced new weaknesses into the function being repaired? Bypass testing and regression measurement are orthogonal properties, and neglecting the latter leaves a material blind spot in any assessment of patch quality.This paper presents an empirical study of vulnerability regression in LLM-generated security patches, applying differential static analysis to 479 complete C/C++ functions spanning eight CWE types. Patches are generated from three frontier models alongside developer-authored ground-truth fixes, for a total of 1,916 analyzed patch–function pairs. AI models fix targeted vulnerabilities at rates approaching human developers, yet they leave a meaningfully larger share of patches in a structurally neutral state, modifying code without engaging its underlying weakness. Across all three AI models, regression rule sets are more similar to each other than to human regression patterns, a consistency specific enough to target with shared automated validators. Finally, for human developers, successfully fixing a vulnerability is strongly associated with lower regression risk. However, this association is absent and not statistically detectable for AI patchers, a distinction with direct implications for how AI patch quality should be verified.
Large language models (LLMs) increasingly repair software vulnerabilities, but most evaluations judge only similarity to a developer fix or removal of the weakness. Neither reveals whether working code was broken. We evaluate eight commercial and open LLMs on 922 JavaScript vulnerability patches, scoring neutralisation and functional preservation. Lacking tests, we score at scale with a reference-based LLM judge, calibrated against execution on a 144-patch benchmark and 254 Java-CVE patches, plus a cross-family judge. The best model fixes 23% of vulnerabilities (judge-based), and cost-efficiency inverts the accuracy ranking. Our central finding concerns the instrument: both judges flag more over-fixes than execution confirms (precision 5–10%), yet on the functional axis agree far more with each other (κ=0.75) than with execution (κ≤0.26), so judge–judge agreement measures reliability, not validity. On real Java code the over-reporting persists, while the judges’ correctness estimates diverge, leaving no single judge trustworthy. Over-fixing is genuine but, under execution, uncommon: a few percent of vulnerability-removing patches, or under 2%, excluding one artefact-prone scenario, both lower bounds. Only adequately tested execution measures the functional-preservation rate, so security-patch evaluation must run the code, use a judge only to rank models, and weigh costs. We release the harness and executable benchmark.
Patrick Deininger, Wolfgang Slany· Journal of Cybersecurity and...· 0 citations
This work presents Porting Benchmark, a curated dataset of 1,234 security patch backporting cases spanning cross-version, cross-branch, and cross-repository scenarios, paired with a common evaluation framework and identifies four root-cause categories (missing target API awareness, cross-version semantic mismatch, non-local dependency propagation failure, and patch construction or localization failure) and derive concrete directions for next-generation tool design.
Jincheng Yang, Yulong Fu, Chengwei Liu et al.· 0 citations
The risk is not tied to any particular model but rather to the task category, and organizations should ask not which tool to trust, but instead whether LLM-generated automation code should be deployed without review.
Shanna M. Kahn, John D. Hastings· arXiv.org· 0 citations
: Large language models are increasingly being deployed in safety-critical domains, yet remain vulnerable to jailbreak attacks that circumvent safety alignments. This systematic review synthesizes empirical jailbreak research published between 2024 and 2025, using a PRISMA-guided search protocol, followed by BERTopic-based topic modeling. The analysis identifies eight main jailbreak categories: optimization-based, ge-netic/evolutionary, iterative refinement, semantic/persuasion-based, decomposition, context/generation-level, visual/encoding and fuzzing attacks, and characterizes their effectiveness, efficiency, and transferability across open-source and proprietary models, including Llama-2/3, Vicuna, GPT-3.5/4, Claude, Gemini, and DeepSeek-V3. Results show that simple configuration and context-level attacks can match the near-perfect attack success rates of sophisticated white-box optimization methods on models such as Llama-2, while requiring far fewer queries and no parameter access, highlighting a gap between research focus and practical threat severity. The review further identifies five recurring vulnerability mechanisms: representation-level gaps, execution-priority manipulation, semantic fragmentation, gradient-space exploitation and persuasion susceptibility, and documents family-specific vulnerability patterns, with open-source Llama-based models consistently more exposed than safety-enhanced architectures such as Claude. Diverse methods, uneven focus on models and publication bias limit how broadly results apply. Nonetheless, the review reveals that weaknesses in safety alignment persist across successive LLM generations, urging that effective defenses must address all eight attack categories rather than isolated techniques.
Meda Račaitytė, Hélder Bastos, R. Ribeiro et al.· International Conference on...· 0 citations
PRGuard is introduced, an attributable PR security reviewer that constructs candidate vulnerabilities and validates their premises against repository evidence using deterministic, non-executing tools and bounded retrieval and uncovers twelve previously undisclosed, proof-of-concept-backed vulnerabilities.
Zhuo Chen, Boyang Wang, Xi-Yue Zhang et al.· 0 citations
Security patches (SPs) are the main mechanism for fixing software vulnerabilities, yet a single vulnerability is not always resolved by a single patch: fixes may be completed incrementally, propagated across maintained branches, or replicated across related repositories. When patch records are incomplete, downstream users may observe only part of the required fix set and therefore apply only partial patching. However, comprehensive patch discovery remains difficult because the prevalence and causes of the multi-SP phenomenon are still poorly understood. In this paper, we present the first large-scale empirical study of multi-SP vulnerabilities. By merging four major vulnerability databases, we construct a dataset of 6,053 multi-SP CVEs with 16,260 SPs, showing that 20.6% of CVEs with patches involve multiple SPs and that merging databases increases recognized multi-SP CVE counts by 36-55% over any single source. We further analyze why a vulnerability is associated with multiple SPs and derive a two-level taxonomy with 6 categories and 16 sub-categories. Based on these findings, we develop SPectre, a taxonomy-driven prototype for comprehensive patch discovery. On 300 multi-SP CVEs, after manually verifying ground-truth SPs, SPectre improves multi-SP patch coverage over representative patch localization baselines, achieving 0.927 recall on same-repository cases and 0.873 recall on cross-repository cases after manual ground truth verification. On 100 recent CVEs recorded as single-patch by all public databases, SPectre further discovers 28 previously unreported SPs across 20 CVEs. Our results show that multi-SP vulnerabilities are both prevalent and systematically underreported, motivating stronger patch-completeness awareness, improved vulnerability database curation, and relation-aware security tooling.
Unknown authors· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
