Jul 2026· Signal Processing and Communications Applications Conference· pp. 1-4· 0 citations· 18 references
Abstract
The rapid advancement of Large Language Models (LLMs) has facilitated their ubiquitous integration into various domains, leading to widespread adoption. However, this escalating trend has introduced significant security vulnerabilities, necessitating the identification and mitigation of flaws arising from malicious exploitation. Red teaming assessments, conducted to evaluate model robustness through diverse adversarial inputs, are essential for exposing security risks and implementing countermeasures. Currently, red teaming is performed either manually by experts or automatically using predefined attack datasets. Nevertheless, manual testing remains time-consuming, while existing automated methods suffer from limited creativity due to their inherent dependency on fixed datasets. In this study, we propose an automated, human-independent, and adaptive approach leveraging GFlowNets to identify LLM vulnerabilities by utilizing one large language model to test another. Within this framework, an attacker model is trained against a specified victim model to perform automated red teaming and provide a quantitative robustness score. This research aims to generate more effective adversarial attacks in English compared to existing benchmarks and, as a novel contribution to the literature, introduces a model capable of generating attack inputs in the Turkish language.
This study proposes an automated, human-independent, and adaptive approach leveraging GFlowNets to identify LLM vulnerabilities by utilizing one large language model to test another, and introduces a model capable of generating attack inputs in the Turkish language.
Berkay Ozcam, Irem Onen, M. Amasyalı et al.· 0 citations
This survey provides a comprehensive analysis of adversarial prompting strategies, ranging from input manipulation techniques to semantic and structural distortions, and explores defense strategies across preprocessing, model-level, postprocessing, and hybrid strategies, highlighting recent advances and their limitations.
A. Nasution, Ahmet Emre Ergün, Aytu˘g Onan et al.· IEEE Access· 0 citations
The fast deployment of large language models (LLMs) in critical-domain applications such as financial systems has introduced new types of security vulnerabilities (e.g., adversarial prompt injections, semantic obfuscation, and multi-stage social engineering attacks) that conventional countermeasures cannot typically handle. These vulnerabilities need to be detected through the natural language interface interactions, and relevant security policies need to be applied based on the context of the interactions. In this paper, we present a novel Adaptive LLM Threat Response (ALTR) framework, that supports trustworthy decision-making in LLM-enabled applications by integrating real-time behavioral anomaly detection with context-aware prompt validation to mitigate the risks of vulnerable LLMs. Specifically, we develop a Behavioral Anomaly Engine (BAE) that uses Isolation Forests to identify anomalies in user behavior during LLM interactions in financial systems. In addition, we develop a Context-Aware Prompt Classifier (CAPC) that employs DistilBERT and syntactic analysis to determine malicious inputs. To address the latency requirements in threat response, we create a Threat Response Memory (TRM) knowledge base that stores the aggregated behavioral and semantic risk signals from prior user sessions, and uses LSTM to perform longitudinal threat analysis to predict threat severity. We perform validation experiments using public datasets and a taxonomy of AI-based threats outlined in the Cisco AI Defense Capture-the-Flag (CTF) challenge. Our experiment results show the effectiveness of our ALTR approach by achieving an accuracy of 0.961, an AUC of 0.961, and a low false-negative rate of 0.9%, while maintaining an inference latency ($<\mathbf{2 0 m s}$) of across a range of LLM security vulnerabilities.
Vani Seth, Srikar Alla, Sd Md Abidar Rahman et al.· International Conferences on...· 0 citations
The developing cybersecurity threats or large-scale attack vectors, coupled with the rising use of artificial intelligence (AI), are compelling researchers and practitioners to unite their forces in combating these issues through coordinated efforts. This paper investigates the use of LLMs developed with MITRE ATT&CK techniques, an established knowledge base that describes an adversary’s tactics, techniques and procedures based on real cyber operations. The experiment explores utilizing LLMs to generate code snippets, implementation ideas, and descriptive summaries corresponding to ATT&CK techniques to assist security researchers, educators, penetration testers, and red teams in developing a better understanding of adversarial behavior and defensive strategy. These abilities can enhance training, hasten security analyses, and create more credible evaluation testing grounds for detection and response mechanisms. Simultaneously, the availability of automated tools to code generators could raise a whole set of security issues, as they might lower the level of expertise required to replicate an offensive technique. Given these observations, careful governance, responsible application and proper safeguards ought to be exercised while applying LLMs in the domain of cybersecurity. By discussing the benefits and risks associated with LLMs, this paper attempts to present both sides of the coin in terms of the opportunities and challenges they present in relation to MITRE ATT&CK and modern cyber defences.
Sameeruddin Shaik· International Journal of Int...· 0 citations
As large language models (LLMs) are deployed in security-critical environments, evaluating their robustness against adversarial threats is essential. We present a two-phase evaluation of ten Llama variants using the OWASP Top 10 for LLM Applications. In Phase I we benchmark 100 OWASP-based adversarial prompts; Llama-Guard-3-1B leads (76% detection, 0.165 s latency, 0.94 GB VRAM), while base models detect nothing. In Phase II we apply nine encoding obfuscations to the same prompts (9,000 inferences): Base64 stays detectable (up to 100%), hexadecimal collapses to 2%, and ASCII art fully bypasses all text-only models. Model scale alone does not ensure robustness; targeted safety training and encoding-aware defenses are required. Both benchmark datasets are released for reproducibility [3].
Nourin Shahin, I. Alsmadi· Practice and Experience in A...· 0 citations
: Large language models are becoming pervasive core components in many real-world applications. As a consequence, security alignment represents a critical requirement for their safe deployment. Although previous related works focused primarily on model architectures and alignment methodologies, these approaches alone cannot ensure the complete elimination of harmful generations. This concern is reinforced by the growing body of scientific literature showing that attacks, such as jailbreaking and prompt injection, can bypass existing security alignment mechanisms. As a consequence, additional security strategies are needed both to provide qualitative feedback on the robustness of the obtained security alignment at the training stage, and to create an “ultimate” defense layer to block unsafe outputs possibly produced by deployed models. To provide a contribution in this scenario, this paper introduces SecureBreak , a safety-oriented dataset designed to support the development of AI-driven solutions for detecting harmful LLM outputs caused by residual weaknesses in security alignment. The strong reliability of the proposed dataset derives from the adopted manual annotation procedure, in which labels are assigned conservatively to prioritize safety even in the presence of minor disagreements in the annotators’ opinion. Our exploratory data analysis campaign shows satisfactory performance in the detection of unsafe content across several risk categories. To evaluate its effectiveness, we measure the performance of several pre-trained LLMs in the considered classification setting under baseline conditions and compare these results with those obtained after fine-tuning the same models on SecureBreak . The results indicate that the dataset is valuable not only for constructing post-generation filtering modules that act as a last-line defense, but also for building additional supervisory intelligence for alignment optimization. We show that models fine-tuned on SecureBreak improve safety classification by up to 20% over baselines, with small models reaching up to 90.14% accuracy and sometimes outperforming larger models. In particular, classifiers derived from SecureBreak can be used to measure residual safety failures, inform whether additional training or refinement steps are necessary, and ultimately support more controlled and effective security alignment workflows.
Marco Arazzi, Vignesh Kumar Kembu, Antonino Nocera· Proceedings of the 15th Inte...· 0 citations