EGP-Defense: Enhancing Adversarial Robustness of LVLMs via Training-Free Edge-Guided Prompting

Abstract

Large Vision-Language Models (LVLMs) have demonstrated remarkable multimodal comprehension capabilities, achieving state-of-the-art performance across various vision-language tasks. However, their performance drops significantly when facing adversarial attacks on the visual encoder. To alleviate this issue, existing approaches often rely on adversarial training, enhancing model robustness through substantial computational cost. Unlike these methods, this article proposes a novel, training-free adversarial defense method called Edge-Guided Prompt Defense (EGP-Defense), which performs adversarial defense during the model inference stage. This method is based on a comprehensive analysis of image edges under various types of attacks. We observe that edge maps exhibit strong robustness against adversarial attacks, and the extracted edge features can effectively reflect key aspects of the original image. Building on this observation, we first apply the Canny operator to extract edge maps from input images, and then use LVLMs to generate textual descriptions based on these structural representations. To further distill the most critical information from these descriptions, we extract informative keywords and incorporate them as auxiliary prompts. These prompts guide the model to focus on task-relevant features during inference, thereby enhancing its robustness against adversarial perturbations. Extensive experiments demonstrate that EGP-Defense significantly improves the robustness of LVLMs against three types of adversarial attacks in both image classification and image caption tasks. Our code is available at https://github.com/wangboyu0414-svg/EGP-Defense.