Enhancing security in LLM applications: a performance evaluation of early detection systems

Abstract

Abstract Prompt injection (PI) attacks threaten novel software applications, which have LLM-based functionality. Prompt leakage, a variant of prompt injection, constitutes a serious confidentiality risk to those applications. Existing defenses against PI attacks currently cannot identify prompt leaks precisely. Moreover, an attacker can construct leakage attacks, which could evade most of these defenses. This prevents the ubiquitous adoption of LLMs in software applications. Meanwhile, there is a lack of practical investigations into the effectiveness of prompt injection (PI) detection tools. There is a gap in practical knowledge on how precisely existing tools detect PI attacks, and how they should be further improved. We evaluated the capabilities of early prompt injection detection systems, focusing on the performance of detection techniques implemented in several open-source solutions. We tested the solutions against prompt leak attacks that employed widespread injection techniques such as context-ignoring and context-manipulation. We present an analysis of distinct PI detection techniques and a comparative analysis of LLM Guard, Vigil, and Rebuff. We concluded that the designs of canary word-based detection techniques in Vigil and Rebuff were weak against our prompt leak attacks. We propose improvements for them. We found an evasion weakness in Rebuff’s secondary model-based technique and proposed a mitigation. We revealed that, thanks to their detection policies, Vigil is optimal for cases when a minimal false positive rate is required, and Rebuff is the most optimal for the highest detection rate.