This study proposes an automated, human-independent, and adaptive approach leveraging GFlowNets to identify LLM vulnerabilities by utilizing one large language model to test another, and introduces a model capable of generating attack inputs in the Turkish language.
Abstract
The rapid advancement of Large Language Models (LLMs) has facilitated their ubiquitous integration into various domains, leading to widespread adoption. However, this escalating trend has introduced significant security vulnerabilities, necessitating the identification and mitigation of flaws arising from malicious exploitation. Red teaming assessments, conducted to evaluate model robustness through diverse adversarial inputs, are essential for exposing security risks and implementing countermeasures. Currently, red teaming is performed either manually by experts or automatically using predefined attack datasets. Nevertheless, manual testing remains time-consuming, while existing automated methods suffer from limited creativity due to their inherent dependency on fixed datasets. In this study, we propose an automated, human-independent, and adaptive approach leveraging GFlowNets to identify LLM vulnerabilities by utilizing one large language model to test another. Within this framework, an attacker model is trained against a specified victim model to perform automated red teaming and provide a quantitative robustness score. This research aims to generate more effective adversarial attacks in English compared to existing benchmarks and, as a novel contribution to the literature, introduces a model capable of generating attack inputs in the Turkish language.
The rapid advancement of Large Language Models (LLMs) has facilitated their ubiquitous integration into various domains, leading to widespread adoption. However, this escalating trend has introduced significant security vulnerabilities, necessitating the identification and mitigation of flaws arising from malicious exploitation. Red teaming assessments, conducted to evaluate model robustness through diverse adversarial inputs, are essential for exposing security risks and implementing countermeasures. Currently, red teaming is performed either manually by experts or automatically using predefined attack datasets. Nevertheless, manual testing remains time-consuming, while existing automated methods suffer from limited creativity due to their inherent dependency on fixed datasets. In this study, we propose an automated, human-independent, and adaptive approach leveraging GFlowNets to identify LLM vulnerabilities by utilizing one large language model to test another. Within this framework, an attacker model is trained against a specified victim model to perform automated red teaming and provide a quantitative robustness score. This research aims to generate more effective adversarial attacks in English compared to existing benchmarks and, as a novel contribution to the literature, introduces a model capable of generating attack inputs in the Turkish language.
Berkay Özçam, İrem Önen, E. I. Tatli et al.· Signal Processing and Commun...· 0 citations
This survey provides a comprehensive analysis of adversarial prompting strategies, ranging from input manipulation techniques to semantic and structural distortions, and explores defense strategies across preprocessing, model-level, postprocessing, and hybrid strategies, highlighting recent advances and their limitations.
A. Nasution, Ahmet Emre Ergün, Aytu˘g Onan et al.· IEEE Access· 0 citations
The fast deployment of large language models (LLMs) in critical-domain applications such as financial systems has introduced new types of security vulnerabilities (e.g., adversarial prompt injections, semantic obfuscation, and multi-stage social engineering attacks) that conventional countermeasures cannot typically handle. These vulnerabilities need to be detected through the natural language interface interactions, and relevant security policies need to be applied based on the context of the interactions. In this paper, we present a novel Adaptive LLM Threat Response (ALTR) framework, that supports trustworthy decision-making in LLM-enabled applications by integrating real-time behavioral anomaly detection with context-aware prompt validation to mitigate the risks of vulnerable LLMs. Specifically, we develop a Behavioral Anomaly Engine (BAE) that uses Isolation Forests to identify anomalies in user behavior during LLM interactions in financial systems. In addition, we develop a Context-Aware Prompt Classifier (CAPC) that employs DistilBERT and syntactic analysis to determine malicious inputs. To address the latency requirements in threat response, we create a Threat Response Memory (TRM) knowledge base that stores the aggregated behavioral and semantic risk signals from prior user sessions, and uses LSTM to perform longitudinal threat analysis to predict threat severity. We perform validation experiments using public datasets and a taxonomy of AI-based threats outlined in the Cisco AI Defense Capture-the-Flag (CTF) challenge. Our experiment results show the effectiveness of our ALTR approach by achieving an accuracy of 0.961, an AUC of 0.961, and a low false-negative rate of 0.9%, while maintaining an inference latency ($<\mathbf{2 0 m s}$) of across a range of LLM security vulnerabilities.
Vani Seth, Srikar Alla, Sd Md Abidar Rahman et al.· International Conferences on...· 0 citations
This study systematically examines the vulnerabilities of open and closed-weight LVLMs, including LLaVA, InstructBLIP, Fuyu, Qwen, DeepSeek, Gemini, GPT, and Grok, using adversarial prompting strategies informed by social theories to simulate real-world social manipulation tactics.
Abdulkadir Erol, Trilok Padhi, Agnik Saha et al.· ACM Transactions on Intellig...· 0 citations
This paper investigates the use of LLMs developed with MITRE ATT&CK techniques to generate code snippets, implementation ideas, and descriptive summaries corresponding to ATT&CK techniques to assist security researchers, educators, penetration testers, and red teams in developing a better understanding of adversarial behavior and defensive strategy.
Sameeruddin Shaik· International Journal of Int...· 0 citations
SecureBreak is introduced, a safety-oriented dataset designed to support the development of AI-driven solutions for detecting harmful LLM outputs caused by residual weaknesses in security alignment and is valuable not only for constructing post-generation filtering modules that act as a last-line defense, but also for building additional supervisory intelligence for alignment optimization.
Marco Arazzi, Vignesh Kumar Kembu, Antonino Nocera· Proceedings of the 15th Inte...· 0 citations