Target-Oriented Adversarial Patches Against Vision–Language Models via Heatmap-Guided Localization

Abstract

Vision-language models (VLMs) are increasingly used in safety-related settings; therefore, it is vital to understand their underlying vulnerability to adversarial manipulation. Adversarial patches are practical local perturbations. However, targeted patch attacks on Contrastive Language-Image Pre-training (CLIP) remain insufficiently understood under strict spatial budgets and realistic imaging variation. Accordingly, we propose a target-oriented adversarial patch framework that addresses both patch placement and patch optimization. For patch placement, we introduce heatmap-guided localization including Random, Target Saliency, Gradient-weighted Class Activation Mapping (GradCAM), and Grad-Attention, and compare their performance. For patch optimization, we combine target cross-entropy, target-text feature alignment, competing-class suppression, original-class suppression, and total-variation regularization with an expectation-over-transformation (EOT) strategy. During optimization, random transformations such as rotation, perspective, color changes, and noise are applied to improve patch stability under varying imaging conditions. Experiments on OpenAI CLIP, OpenCLIP, MetaCLIP, and EVA-CLIP show that the three semantic guidance strategies improve attack success over random placement, and among them, Grad-Attention performs best on average because it aligns more closely with the model’s internal attention structure. Ablation studies on patch size, loss composition, and target class further show that cross-entropy is the most critical loss term and semantically distinctive targets are easier to induce. Furthermore, cross-model evaluation indicates that patches exhibit non-trivial transferability within the CLIP family, particularly when EVA-CLIP is used as the source model.