HoF-Bench (named after AISLE's public Hall of Fame), a benchmark built from 95 public AI-discovered CVEs across eight repositories pinned at vulnerable commits, provides a compact test bed for comparing vulnerability scanners, their reliability across repeated runs, and the candidate volume they create.
Abstract
LLM-based analyzers have begun finding real vulnerabilities in mature open-source projects: AISLE's analyzer is credited with more than 280 CVEs across 78 projects, including OpenSSL, curl, and GnuTLS. We introduce HoF-Bench (named after AISLE's public Hall of Fame), a benchmark built from 95 of these public AI-discovered CVEs across eight repositories pinned at vulnerable commits. Analyzers receive source and target-file scope but not CVE identifiers, descriptions, fixes, or expected mechanisms; a detector-blinded frontier-model judge credits only findings that identify the same code path, root cause, attack condition, and impact. A deliberately minimal LLM-based analyzer rediscovers up to 65 of the 95 CVEs (68%) under this strict protocol. No frontier model performs detection anywhere in the study. The ten detector backbones are five open-weight models (21B--284B total parameters, 3--13B active) and five proprietary small or"flash"-tier models. All of them run in the fixed scaffold with four repeated passes, an optional generated-context stage, and a replayable multi-round triage stage (7,600 model--CVE pass records). Difficulty is strongly structured by language; the CVEs missed by every model concentrate in C infrastructure code. HoF-Bench provides a compact test bed for comparing vulnerability scanners, their reliability across repeated runs, and the candidate volume they create. The dataset is available at https://huggingface.co/datasets/aisleinc/HoF-Bench.
The results show that access to repository history is insufficient: concealment becomes most effective when benign and malicious changes jointly occupy the auditor's active review context or when the stated purpose plausibly accounts for the attack-bearing diff.
Candidate generation for open-source vulnerabilities is no longer scarce. AI-assisted code review now produces defect candidates cheaply, and industry programs pair them with expert human triage. The remaining scarcity is validation and impact assessment, and the gap is largest in critical-infrastructure software like 5G cores. Here, validation has four costs: verification, reachability, impact, and fix-completeness. We present Drishti, an AI-led human-directed vulnerability audit framework with four components, one per cost: (i) an anti-pattern catalog for verification, (ii) critical-path triage for reachability, (iii) concentric validation for impact, and (iv) patch-review for fix-completeness. Across audits of Open5GS and free5GC, Drishti produced three findings. The first is a pre-authentication NULL-dereference in the Open5GS NRF multipart parser, fixed upstream with a CVE requested. The second is an ASN.1-PER memory amplification in the free5GC NGAP decoder. A 2-byte input from a rogue gNodeB OOM-kills the AMF in 6.2 seconds. The third is a defective patch on CVE-2025-69248 whose defense-in-depth check is dead code before authentication.
Evaluating the ability of large language models (LLMs) to discover software bugs is increasingly important. Existing benchmarks typically evaluate this capability by asking the model to generate a proof-of-concept input that triggers a predefined target vulnerability. However, this setup may overlook valid crashes discovered by the model when they do not match the predefined target. As a result, the evaluation may not reflect the model's real capability. We present FuzzingBrain-Bench, a benchmark for assessing AI models'ability to discover bugs in open-source software. Models are given an open-source project and a sanitizer-instrumented harness in a self-contained Docker image. Their goal is to generate inputs that trigger as many distinct crashes as possible through the harness. A model's performance on each challenge is scored based on the number of distinct crash signatures it produces, capped at a predefined maximum and weighted by a difficulty coefficient. FuzzingBrain-Bench V1 consists of 77 challenges drawn from 43 open-source projects, with 36 C, 32 C++, and 9 Java/JVM challenges. We evaluate Claude Haiku 4.5, Claude Sonnet 4.6, and Claude Opus 4.8 on the full benchmark. Claude Opus 4.8 performs best, triggering crashes in 60 of 77 challenges and achieving a score of 196 out of 579. None of the three models triggers a crash in 13 challenges. The FuzzingBrain-Bench corpus and harnesses are publicly available at https://github.com/fuzzingbrain/FuzzingBrain-Bench.
Ze Sheng, Aleksandar Kezic, Zhicheng Chen et al.· 0 citations
: Large language models are increasingly being deployed in safety-critical domains, yet remain vulnerable to jailbreak attacks that circumvent safety alignments. This systematic review synthesizes empirical jailbreak research published between 2024 and 2025, using a PRISMA-guided search protocol, followed by BERTopic-based topic modeling. The analysis identifies eight main jailbreak categories: optimization-based, ge-netic/evolutionary, iterative refinement, semantic/persuasion-based, decomposition, context/generation-level, visual/encoding and fuzzing attacks, and characterizes their effectiveness, efficiency, and transferability across open-source and proprietary models, including Llama-2/3, Vicuna, GPT-3.5/4, Claude, Gemini, and DeepSeek-V3. Results show that simple configuration and context-level attacks can match the near-perfect attack success rates of sophisticated white-box optimization methods on models such as Llama-2, while requiring far fewer queries and no parameter access, highlighting a gap between research focus and practical threat severity. The review further identifies five recurring vulnerability mechanisms: representation-level gaps, execution-priority manipulation, semantic fragmentation, gradient-space exploitation and persuasion susceptibility, and documents family-specific vulnerability patterns, with open-source Llama-based models consistently more exposed than safety-enhanced architectures such as Claude. Diverse methods, uneven focus on models and publication bias limit how broadly results apply. Nonetheless, the review reveals that weaknesses in safety alignment persist across successive LLM generations, urging that effective defenses must address all eight attack categories rather than isolated techniques.
Meda Račaitytė, Hélder Bastos, R. Ribeiro et al.· International Conference on...· 0 citations
With the increasing usage of AI-generated code in software development workflows, new security challenges and concerns arise. This paper analyzes five LLMs: ChatGPT, Claude, Gemini, DeepSeek, and Grok in three phases of security assessments against web vulnerabilities listed by the OWASP Top 10. Phase 1 (December 2025) evaluated 13 prompts in XSS, Authentication, API Security, Hardcoded Secrets, and React Server Component CVE. Phase 2 (March 2026) evaluated 11 prompts using newly disclosed CVEs, complex authentication, and API security, with updated models. Phase 3 (May 2026) revisited phase one prompts against current models. Across all 185 samples, 49 out of the 120 Phase 1 and 2 samples (40.8%) were identified as vulnerable, 65 (54.2%) as secure and 6 (5.0%) partially vulnerable. The CVE-based prompts recorded the highest percentage of vulnerability (76%). No failures were observed in the authentication prompts with simple patterns, while complex access control resulted in 45% failures. Phase 3 (65 samples) re-ran Phase 1 prompts on current models; no progress was observed; the partial result rate went up from 7.7% to 27.7%. The results revealed hedged responses from the updated models. All results were cross-validated using Semgrep static analysis.
Malak Mansour, Anas AlMajali· IEEE Jordan Conference on Ap...· 0 citations
GenIaC-SecBench is introduced, a benchmark of 100 deployment scenarios stratified by architectural complexity, evaluated across 12 model configurations from four vendors, producing 1,196 IaC artifacts scanned by three independent policy engines (Checkov, Trivy, KICS).
Animesh Shaw· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
