Red Teaming LLMs via Linguistic-Aware Fuzzing

Abstract

Safety alignment aims to prevent Large Language Models (LLMs) from producing harmful content. However, safety alignment remains vulnerable to malicious instructions. Red teaming is a critical methodology for identifying such vulnerabilities in LLMs. Existing approaches often rely on jailbreak templates or rule-based transformation, limiting the diversity of generated tests and the continued testing capability of these test approaches. To address these limitations, we propose Lingfuzz, a linguistic-aware fuzzing framework for continuing red teaming LLMs. The key idea of this framework is to mutate the existing malicious instructions at the lexical and syntactic levels, while keeping the malicious intentions of the instructions. Such mutations enable generating diverse malicious instructions due to the unlimited space of lexical and syntactic choices, while having a continued testing capability by iteratively mutating the mutants. We evaluated Lingfuzz on five aligned commercial LLMs and one open-source LLM in black-box testing. The results show that Lingfuzz triggers safety alignment vulnerabilities in 71.0% of the cases in JailbreakBench benchmark, higher than the second-highest baseline of 63.8%. Lingfuzz also possesses outstanding multilingual testing capabilities that far exceed other baselines with approximately 24% ESR advantage testing in Jailbench benchmark. The malicious instructions generated by Lingfuzz have almost three times higher diversity than previous work according to the self-BLEU metric. Lingfuzz also demonstrates strong continued testing capability by showing five times less sensitivity to the LLM evolution than other approaches.