BERM is introduced, a lightweight framework that performs in-situ detection by modeling a host LLM’s internal representations extracted during prefill, adding negligible overhead and reducing incremental inference overhead to near-zero.
Large language models (LLMs) embedded in enterprise workflows cannot structurally distinguish legitimate instructions from adversarial ones in the same token stream, making prompt injection OWASP's top LLM risk for two consecutive editions a persistent threat across direct and indirect vectors. This paper presents PromptShield-RT, a layered, real-time, model-agnostic framework combining input normalization and provenance tagging, lexical-heuristic pattern matching, a statistical classifier, structural anomaly features, and calibrated risk fusion, with policy-driven mitigation (allow/sanitize/quarantine/block) and an explainable, adaptive-feedback mechanism for SOC workflows. We construct an original evaluation corpus, SynPI-Bench (n = 450, six categories), and a template-disjoint held-out generalization set (n = 31) with novel phrasings, obfuscation encodings, and adversarial hard-negative benign text. Using template-grouped 5-fold cross-validation, the fused pipeline achieves 92.4% accuracy (F1 = 0.930, AUC = 0.990), outperforming heuristic-only (57.0%) and naive-averaged (59.2%) baselines, while a lexical classifier reaches 85.9% with lower precision. We report a pronounced generalization gap on the held-out set (48.4% accuracy, 90% false-positive rate on hard negatives), quantifying a known limitation of surface-lexical defenses. The pipeline achieves sub-millisecond P95 latency (0.266 ms), within typical 50 ms enterprise SLAs. We situate PromptShield-RT relative to structural, architectural, and guardrail-product defenses, arguing for layered, defense-in-depth architectures, with reproducible code provided.
Fatimah Alhamzawi· Al-Noor Journal of Engineeri...· 0 citations
Large language models (LLMs) are now embedded in many digital systems, which has exposed a new and distinctive attack surface. The most pressing threat is the prompt injection attack, in which adversarial text causes a model to ignore its developer-defined instructions, leak its hidden system prompt, or perform unintended actions. This paper proposes a layered hybrid defense gateway that combines three complementary detectors: a fast prefilter based on lexical rules, a semantic classifier that uses word-level TF-IDF features together with a multilayer perceptron, and a character-level classifier that is robust to obfuscation. Their outputs are merged by a lightweight meta-decision block based on logistic regression. In a fully reproducible experiment, the detectors are trained on clean and standard obfuscated attacks and evaluated on held-out data that additionally contains unseen, newly phrased attacks and an unseen adaptive obfuscation style. On this controlled benchmark the proposed hybrid system reaches an F1 score of 0.989 and the lowest false-positive rate among all compared systems, clearly outperforming a brittle rule-based baseline (F1 = 0.504) and every single classifier across all attack categories. An ablation study confirms that each layer covers a distinct class of attacks, empirically supporting the defense-in-depth principle.
O. Esanov, Ch. Erkinova· 2022-yil 3-son (133/1) ANIQ...· 0 citations
Multimodal Retrieval Augmented Generation (M-RAG) is increasingly vulnerable to adversarial attacks where malicious data are crafted to produce embeddings that align with benign entries in the vector space, deceiving retrieval and inducing harmful outputs. Existing defenses primarily operate at query time, relying on auxiliary detectors, similarity re-ranking, or feature-consistency checks. However, these approaches suffer from non-trivial inference overhead, generalize poorly to unseen attack strategies, and often assume specific attack distributions. To address this, we propose DSPrompt, a Dynamic Soft Prompt defense framework that directly reshapes the retriever's embedding semantics, without modifying the retrieval pipeline. It inserts few learnable soft prompts into each layer of the visual and textual encoders of a frozen retriever, utilizing a shallow-to-deep length schedule that is adaptive to the capacity in the model layers. These prompts are trained under a dynamic min-max scheme: an online multimodal attacker continually crafts hard adversarial documents against the current retriever, while the defender is updated to push such documents out of the top-k while preserving the ranking and diversity of benign evidence. Because the defended encoder can be pre-computed and indexed exactly as in standard dense retrieval, DSPrompt incurs no additional per-query optimization and introduces fewer than 1% additional parameters. Extensive experiments across four benchmarks and three representative poisoning attacks show that DSPrompt substantially reduces the attack success rate and poison retrieval rate while maintaining near-lossless retrieval utility and generation fidelity, consistently outperforming existing defense baselines at a fraction of their computational cost.
Chang Liu, Y. Lai, Mingyue Cui et al.· 0 citations
Prompt injection is a critical security threat in large language model (LLM) applications, where attackers hijack model behavior by embedding malicious instructions in user or external data. Existing detection methods only detect the presence of injection and refuse to respond upon detection, overlooking the fact that for many modern aligned models, well-crafted instructions can resist most injection attacks. This means that the injection robustness varies significantly across instructions and models. This leads to widespread unnecessary over-refusal: inputs containing injections that the model could have handled correctly are rejected incorrectly. To deal with this over-refusal issue, we propose BASIS (Robustness-Aware Prompt Injection Defense). This defense method uses the Attention Competition Ratio ($\rho$) as features to train two sparse linear probes: an existence probe and a breach probe. Both probes make defense decisions through cascaded gating, which does not require additional LLM inference. BASIS comprises three stages: injection existence detection, per-sample breach prediction, and instruction robustness assessment; the online cascade refuses only when the model would actually be compromised and thus avoids over-refusal on robust instructions. Experiments across four tasks and six open-source LLMs show that BASIS maintains near-perfect injection detection while substantially reducing over-refusal on safe attack samples, especially under robust instruction templates.
Laiqiao Qin, Tianqing Zhu, Longxiang Gao et al.· 0 citations
Large language models (LLMs) are vulnerable to backdoor attacks, where hidden triggers induce malicious outputs. Existing defenses generally fall into inference-time detection or training-time mitigation, but face two key limitations. First, they focus on fine-tuning-based backdoors (e.g., PEFT modules) and fail to address insidious model-editing attacks that bypass training pipelines. Second, they target simple classification settings and do not naturally extend to open-ended LLM generation and do not naturally extend to the open-ended generation characteristics of LLMs. Consequently, these methods focus on surface-level behavioral patterns while neglecting the deeper representational causes of malicious activations. This lack of mechanistic understanding forces defenses to depend on empirical heuristics, limiting their robustness, generality, and practical applicability in real-world LLM deployment. To bridge this gap, we introduce DeCNIP (Defense with Critical Neuron Isolation Pruning), which leverages representational analysis to identify and neutralize backdoors in a unified pipeline. Specifically, DeCNIP identifies trigger-like behaviors by optimizing a cross-entropy loss between harmful prompts with candidate tokens and benign inputs. This representational discovery exposes latent threats by uncovering mechanisms through which triggers hijack model weights. It then isolates Backdoor Critical Neurons (BCNs) and prunes them selectively to remove malicious influence while preserving model utility. Extensive evaluations on six open-source LLMs and two benchmark datasets demonstrate that DeCNIP achieves over 95% relative reduction in Attack Success Rate (ASR), outperforming seven state-of-the-art defenses with only 0.1% neuron intervention. Moreover, it maintains 97% of the model's performance on normal benchmarks, demonstrating its efficacy, robustness, and scalability.
Yuxi Li, Zhibo Zhang, Kailong Wang et al.· 0 citations
Modern Software-as-a-Service (SaaS) environments generate voluminous, heterogeneous logs that exhibit rapid schema drift and semantic sparsity, rendering traditional template-based detection methods ineffective. This paper presents LLM-EP, a unified framework that utilizes Large Language Models (LLMs) for proactive error prediction via semantic log analysis. Unlike existing heuristics, LLM-EP employs a drift resilient semantic parser and an embedding driven normalization engine to maintain accuracy amidst evolving log formats. We propose a hybrid offline-online architecture that balances deep semantic understanding with real-time inference requirements, incorporating a robust outof-vocabulary (OOV) mechanism for zero-shot generalization to unseen log patterns. Evaluation on production SaaS datasets indicates that LLM-EP achieves a 25% gain in parsing accuracy and a 12% improvement in F1-score over state-of-the-art baselines. Most notably, LLM-EP reduces Mean Time to Detect (MTTD) by up to 60%, demonstrating its efficacy as a scalable foundation for automated site reliability engineering.