Sep 2025· ACM Transactions on Software Engineering and Methodology· 3 citations· ⚡ 1 influential· 68 references
Computer Science
TL;DR
This initial study demonstrates that directly prompting off-the-shelf code LLMs remains ineffective; even advanced prompting strategies cannot compensate for the lack of task-specific knowledge, and fine-tuning proves highly effective, with pseudo-code representation consistently yielding the best performance.
Abstract
Security patch detection (SPD) is crucial for maintaining software security, as unpatched vulnerabilities can lead to severe security risks. In recent years, learning-based SPD approaches have achieved promising results on source code. However, they are generally not directly applicable to closed-source and proprietary software, which constitute a substantial portion of real-world systems, because such software typically releases patches only in binary form while its source code remains inaccessible. Despite the impressive performance of code large language models (LLMs) on code intelligence and binary analysis tasks, such as decompilation and compiler optimization, their potential for detecting binary security patches remains largely unexplored, revealing a notable gap between their demonstrated low-level code understanding capabilities and this security-critical task. To narrow this gap, we construct a large-scale binary patch dataset comprising 19,448 samples with two representations, assembly code and pseudo-code, and systematically evaluate 19 code LLMs of varying scales for binary SPD under both prompting and fine-tuning settings. Our initial study demonstrates that directly prompting off-the-shelf code LLMs remains ineffective; even advanced prompting strategies cannot compensate for the lack of task-specific knowledge. In contrast, fine-tuning proves highly effective, with pseudo-code representation consistently yielding the best performance. The best fine-tuned model, LLM4Decompile-9B-v2, achieves 0.915 in accuracy, 0.897 in F1 score, and 0.058 in false positive rate on pseudo-code. On average, models fine-tuned on pseudo-code improve accuracy by 27.5% and F1 score by 46.1%, while reducing the false positive rate by 46.2%, relative to their assembly-based counterparts. To understand this advantage, we analyze pseudo-code and assembly code from the perspectives of embedding-space similarity and code naturalness, and find that pseudo-code more closely resembles source code, making it better aligned with the source-code-centric pretraining of code LLMs. Motivated by this finding, we further augment the pseudo-code dataset with source code data, yielding additional gains after fine-tuning, particularly for smaller models. Beyond in-distribution evaluation, we further assess representative fine-tuned code LLMs under deployment-relevant distribution shifts and find that their robustness varies across shift types. Model performance remains stable under decompiler shift and is relatively preserved under architecture shift, suggesting that fine-tuned code LLMs retain meaningful robustness under practical distribution shifts in binary security patch detection.
This work systematically generates syntactic variants of security-relevant code generation prompts and evaluates their impact on code security across multiple open LLMs and programming languages, identifying prompt syntax as a concrete security control surface and providing actionable guidance for reducing vulnerability risk in LLM-assisted development.
Matteo Cicalese, Antonio Della Porta, Stefano Lambiase et al.· arXiv.org· 0 citations
This paper introduces a programming language-agnostic, transferable, adversarial attack that exploits this CLM vulnerability and demonstrates that this attack, even when computed using smaller code embedding models, is highly effective and transferable to larger, closed-source embedding models.
Kaicheng Wang, Liyan Huang, Jesse Thomason et al.· 0 citations
Software security has been a long-standing and prominent topic in both industry and academia. However, with the increasing deployment of smart devices across various architectures, there is now a significant demand for cross-architecture software. For instance, the Heartbleed vulnerability (CVE-2014-0160), classified as CWE-125 (Out-of-bounds Read) and disclosed in the OpenSSL library in 2014, serves as a prime example of a widely-deployed software vulnerability capable of affecting systems across diverse computing environments due to OpenSSL’s broad adoption. More broadly, memory-related vulnerabilities remain a persistent and significant threat, accounting for a substantial proportion of reported CVEs in recent years. Moreover, performing static cross-architecture vulnerability analysis on binary code poses particular challenges due to the variations in instruction sets across different architectures. In this paper, we present a novel static approach, called BinCros, to identify known vulnerabilities across different optimization levels and CPU architectures in binary code. It mainly consists of three phases: 1) Generating ground-truth intermediate representation (IR) data from binary code based on domain knowledge. To address the cross-architecture problem, we consider IR, which is a set of smaller, lower-level instructions that break down, describe, and capture all the effects of an instruction from standard architectures like x86, ARM, and MIPS; 2) Learning good embeddings based on the cross-architecture ground-truth datasets. We employ a code-code learning method to capture the semantic differences and maximize the distribution divergences between vulnerable and non-vulnerable samples; 3) Building a prediction model based on high-level feature representations. To demonstrate the effectiveness of BinCros, we conducted experiments with a series of baselines including single architecture-based techniques, cross-architecture-based techniques, and so on. Experimental results show that BinCros outperforms the baselines by at least 15% in terms of F1-measure in almost all cases. We believe this work will inspire other researchers to consider natural language processing-related techniques for cross-architecture software vulnerability detection.
Shigang Liu, Di Cao, Chao Chen et al.· IEEE Transactions on Informa...· 0 citations
Modern code reuse attacks take full advantage of bloated software. Attackers piece together short sequences of instructions in otherwise benign code to carry out malicious actions. Mitigating these reusable code snippets, known as gadgets, has become one of the prime focuses of attack surface reduction research. While some debloating techniques remove parts of software that contain such gadgets, other methods focus on making them unusable by breaking up chains of them, thereby substantially diminishing the possibility of code reuse attacks. Due to a high number of vulnerabilities, third-party libraries (such as glibc) used to be the main focus of this research but recently the scope of such attacks have enlarged to whole programs and thus, the techniques have emerged that deal with whole applications. Attack surface reduction efforts have typically tried to eliminate such attacks by subsetting (debloating) the application, e.g. via user-specified inputs, configurations, However, such techniques suffer from limitations in soundness, i.e., the software might crash during normal executions on valid inputs, or they may be conservative and leave a large amount of attack surface untackled. In this work, we present a general, whole-program attack surface reduction technique called DeckerPlus that significantly reduces gadgets that are accessible to an attacker during an execution phase (called a deck) and has minor performance degradation. DeckerPlus requires no user inputs and leaves all features intact. It uses static analysis to determine key function sets that should be enabled/disabled at runtime. The runtime system enables these function sets at the specified program points during execution. We implement two extensions to improve the baseline technique utilizing function cloning and inlining. As a result, our baseline framework achieves 70.3% average total gadget reduction on SPEC CPU 2017, 88.5% on GNU coreutils, and 89.0% across the application workloads (nginx, Redis, lighttpd, and xpdf). Function cloning improves these reductions to 75.5%, 89.6%, and 94.2%, respectively, and inlining improves them further to 77.6%, 94.7%, and 97.8%. Overall, the baseline and extensions achieve these reductions with modest runtime overhead: average slowdown is 6.5% on SPEC CPU 2017, nginx averages 1.062x slowdown with baseline DeckerPlus and 1.011x with inlining, Redis throughput is essentially unchanged, lighttpd remains close to baseline throughput, and GNU coreutils overhead is negligible. In addition, we provide a gadget chain-breaking case study, including detailed JOP gadget metrics on both Linux and Windows, and show that our framework breaks the shell-spawning chain in all cases.
Alexandra Hussar, Sharjeel Khan, Chris Porter et al.· ACM Transactions on Privacy...· 0 citations
Large language models (LLMs) have been ex- ploited to generate malware, but the effective- ness of guardrails for code generation secu- rity remains unclear. We introduce CS-Guard, the first benchmark to systematically evalu- ate guardrails for code generation security. It covers 1) text-to-code generation with 1000 high-quality malware-generation prompts, 7 jailbreak attacks, and a novel fictional scenario attack (FSA) that embeds malicious intent in a legitimate fictional software-development sce- nario; and 2) code-to-code generation with 331 code prompts spanning code infilling, code completion, and code translation. We empiri- cally evaluate 9 guardrails across seven LLMs. We find that current guardrails perform poorly against malicious code-generation re- quests: for text-to-code, the average attack success rate (ASR) after jailbreaks reaches about 50% for many guardrails; for code-to- code, average ASR approaches 100% on base LLMs and remains high across many guardrails (14.4% to nearly 100%). Our FSA also achieves ASR close to 100% across many guardrails, raising major reliability concerns for real-world software development. To sup- port future research, CS-Guard uses a modular three-layer guardrail taxonomy that lets devel- opers register guardrails for evaluation. We release the benchmark and data to enable fur- ther community evaluation.
: Large language models are increasingly being deployed in safety-critical domains, yet remain vulnerable to jailbreak attacks that circumvent safety alignments. This systematic review synthesizes empirical jailbreak research published between 2024 and 2025, using a PRISMA-guided search protocol, followed by BERTopic-based topic modeling. The analysis identifies eight main jailbreak categories: optimization-based, ge-netic/evolutionary, iterative refinement, semantic/persuasion-based, decomposition, context/generation-level, visual/encoding and fuzzing attacks, and characterizes their effectiveness, efficiency, and transferability across open-source and proprietary models, including Llama-2/3, Vicuna, GPT-3.5/4, Claude, Gemini, and DeepSeek-V3. Results show that simple configuration and context-level attacks can match the near-perfect attack success rates of sophisticated white-box optimization methods on models such as Llama-2, while requiring far fewer queries and no parameter access, highlighting a gap between research focus and practical threat severity. The review further identifies five recurring vulnerability mechanisms: representation-level gaps, execution-priority manipulation, semantic fragmentation, gradient-space exploitation and persuasion susceptibility, and documents family-specific vulnerability patterns, with open-source Llama-based models consistently more exposed than safety-enhanced architectures such as Claude. Diverse methods, uneven focus on models and publication bias limit how broadly results apply. Nonetheless, the review reveals that weaknesses in safety alignment persist across successive LLM generations, urging that effective defenses must address all eight attack categories rather than isolated techniques.
Meda Račaitytė, Hélder Bastos, R. Ribeiro et al.· International Conference on...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
