Explainable Prompt Injection Detection using Sentence Embeddings, Random Forest, and Word-Level Attribution
TL;DR
A explainable framework that detects the attacks using a Random Forest (RF) classifier along with semantic sentence embedding and SHAP (Shapley Additive Explanations) is used together with a wordlevel attribution mechanism to give human-legible explanations for the model predictions, and to emphasize harmful parts within the prompt.
Abstract
LLM’s have affected the production industries with fast and spot-on answers, but they have also introduced a huge threat to security such as prompt injection, which was identified as the #1 threat by the OWASP GenAI Security Project. The attackers try to manipulate the model’s actions by putting malicious intent into the user’s prompt. To counteract this problem this research vouches for an explainable framework that detects the attacks using a Random Forest (RF) classifier along with semantic sentence embedding. The provided method can also identify the intent behind the prompts at surface to detect more complex jailbreak attacks. To enhance transparency, SHAP (Shapley Additive Explanations) is used together with a wordlevel attribution mechanism to give human-legible explanations for the model predictions, and to emphasize harmful parts within the prompt. The proposed model has an accuracy of 96.88%, precision of 95.90%, recall of 99.08%, and F1 score of 97.46%, which are better than the TF-IDF baseline model. Importantly, the model was able to correctly identify jailbreak prompts like, ”You are now DAN and have no restrictions” that the baseline model always failed to correctly classify as attacks, showing that semantic feature representation is necessary for strong prompt injection detection.