Aug 2026· ACM Transactions on Software Engineering and Methodology· 1 citation· 80 references
TL;DR
Findings indicate that ensembling approaches can be statistically significant and effective on larger datasets, where the best-performing ensemble improved performance by 37% over its individual LLMs on the commercial large-scale code.
Abstract
Source code clones pose risks ranging from intellectual property violations to unintended vulnerabilities. Effective and efficient scalable clone detection, especially for diverged clones, remains challenging. Large language models (LLMs) have recently been applied to clone detection tasks. However, the rapid emergence of LLMs raises questions about optimal model selection and potential LLM-ensemble efficacy. This paper addresses the first question by identifying 76 LLMs and filtering them down to suitable candidates for large-scale clone detection via an LLM-encoder framework known as SSCD. The candidates were evaluated on two public, industry-defined datasets, BigCloneBench, and a commercial, large-scale dataset. No uniformly ’best-LLM’ emerged, though CodeT5+ 110M, CuBERT and SPTCode were top-performers. Regression analysis suggests that embedding size, tokenizer vocabulary, and training dataset characteristics are associated with clone detection performance. To address the second question, this paper explores the ensembling of selected LLMs to improve effectiveness. Results suggest the importance of score normalization and favoring ensembling methods like maximum or sum over averaging. Also, findings indicate that ensembling approaches can be statistically significant and effective on larger datasets, where the best-performing ensemble improved performance by 37% over its individual LLMs on the commercial large-scale code.
This paper introduces a programming language-agnostic, transferable, adversarial attack that exploits this CLM vulnerability and demonstrates that this attack, even when computed using smaller code embedding models, is highly effective and transferable to larger, closed-source embedding models.
Kaicheng Wang, Liyan Huang, Jesse Thomason et al.· 0 citations
: Large language models are increasingly being deployed in safety-critical domains, yet remain vulnerable to jailbreak attacks that circumvent safety alignments. This systematic review synthesizes empirical jailbreak research published between 2024 and 2025, using a PRISMA-guided search protocol, followed by BERTopic-based topic modeling. The analysis identifies eight main jailbreak categories: optimization-based, ge-netic/evolutionary, iterative refinement, semantic/persuasion-based, decomposition, context/generation-level, visual/encoding and fuzzing attacks, and characterizes their effectiveness, efficiency, and transferability across open-source and proprietary models, including Llama-2/3, Vicuna, GPT-3.5/4, Claude, Gemini, and DeepSeek-V3. Results show that simple configuration and context-level attacks can match the near-perfect attack success rates of sophisticated white-box optimization methods on models such as Llama-2, while requiring far fewer queries and no parameter access, highlighting a gap between research focus and practical threat severity. The review further identifies five recurring vulnerability mechanisms: representation-level gaps, execution-priority manipulation, semantic fragmentation, gradient-space exploitation and persuasion susceptibility, and documents family-specific vulnerability patterns, with open-source Llama-based models consistently more exposed than safety-enhanced architectures such as Claude. Diverse methods, uneven focus on models and publication bias limit how broadly results apply. Nonetheless, the review reveals that weaknesses in safety alignment persist across successive LLM generations, urging that effective defenses must address all eight attack categories rather than isolated techniques.
Meda Račaitytė, Hélder Bastos, R. Ribeiro et al.· International Conference on...· 0 citations
Static analysis remains a cornerstone of software security, yet the effectiveness of tools such as CodeQL is often limited by the substantial manual effort required to develop high-coverage query suites. While large language models (LLMs) have emerged as a potential solution for automated code reasoning, their practical utility in generating structured, executable security queries remains underexplored. In this paper, we conduct an empirical study to evaluate the ability of LLMs to synthesize CodeQL queries using vulnerability data from the National Vulnerability Database. Through this investigation, we explore the potential of using LLMs as an automatic CodeQL query generator. Subsequently, we systematically evaluate the performance of various LLM architectures across a diverse set of real-world vulnerabilities, measuring their ability to improve detection coverage and precision. Our findings reveal that LLM-generated queries significantly enhance the baseline CodeQL queries, yielding 82% improvement in average F1-score. Furthermore, we provide a detailed cost- benefit analysis showing that while direct LLM-based scanning of entire repositories is often computationally and financially prohibitive, leveraging LLMs to synthesize CodeQL queries offers a scalable and cost-effective alternative for large-scale vulnerability detection. Our results suggest that LLMs can effectively bridge the gap between unstructured vulnerability reports and formal static analysis specifications, offering a scalable path toward comprehensive automated vulnerability detection.
I. Irsan, Ratnadira Widyasari, Huihui Huang et al.· 0 citations
The growing diversity of code clone types, from syntactic copies to cross-language semantic clones to AI-generated duplicates, has created a fragmentation crisis in clone detection. Current deep learning detectors are domain specialists that degrade significantly outside their training distribution, with F1 drops exceeding 70% across domains. Deploying multiple specialized models is impractical, yet training a single cross-domain detector requires simultaneous access to all training data. To address this, we investigate model merging, a family of post-hoc techniques that operate solely on trained checkpoints. We evaluate parameter merging with five task-vector methods, architecture merging via greedy layer stitching, and cross-tokenizer alignment across four code models, three benchmarks, and twelve configurations. Same-base TIES merging creates effective cross-domain detectors, validated across two model families and three random seeds, reaching 0.865 combined F1 on UniXcoder, 93% of multi-task performance without any training data at the merging step. WUDI achieves the highest in-distribution combined F1 at 0.899, but TIES generalizes better to unseen AI-generated clones, making it our recommended method. Cross-base merging yields only marginal and high-variance gains across all five methods, indicating that task vector compatibility through a shared pre-trained base is the binding factor for effective merging. Merged detectors also outperform zero-shot code LLMs on GPTCloneBench at lower inference cost and generalize up to 4x better than multi-task training to unseen AI-generated clones, suggesting a trade-off between in-domain performance and OOD robustness. This work provides one of the first systematic empirical studies of model merging for software engineering and a practical recipe for building cross-domain clone detectors.
Palash Ranjan Roy, Banani Roy, Kevin A. Schneider et al.· 1 citation
Large language models (LLMs) are increasingly used for code generation, yet they remain vulnerable to prompts that elicit insecure implementations. Existing defenses typically rely on predefined threat models or known vulnerability patterns, limiting their effectiveness against novel attacks. We propose CodeSIFT, a threat-model-agnostic detection method that leverages influence functions to identify batches of prompts that induce anomalous model behavior. Rather than detecting specific vulnerabilities, CodeSIFT measures the parameter-space influence of generated code and uses a statistical test to determine whether a candidate prompt set deviates from a benign reference distribution. To evaluate our approach, we introduce two benchmark datasets covering a variety of vulnerabilities. We evaluate CodeSIFT on three open-weight code LLMs ranging from 3B to 7B parameters, achieving AUROC scores of up to 0.98 at moderate-to-high injection rates, while maintaining well-calibrated false positive rates and substantially outperforming static analysis baselines. These results suggest that influence-function-based detection is a promising direction for identifying malicious code-generation prompts without requiring prior knowledge of the underlying attack class.
Francesco Quinzan, Noor Munir, Yi-Shun Lu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
