Adversarial Robustness in Large Vision-Language Models

Abstract

The recent progress in the vision-language model (VLM) research made it one of the key aspects of artificial intelligence due to joint vision and language processing capabilities required for such tasks as image retrieval, visual question answering, autonomous systems and medical image analysis. However, despite the excellent results obtained with such systems, they still appear to be extremely vulnerable to adversarial perturbations which result in a significant decline in prediction accuracy and reliability due to even subtle input transformations. The purpose of this paper is to provide a systematic method for analyzing the adversarial robustness of vision-language models under both clean and perturbed conditions. The suggested framework is based on Semantic Counterfactual Augmentation (SCA) and Curriculum Contrastive Adversarial Training (CCAT) and aims to maintain semantic consistency and increase the robustness of the model respectively. The framework is tested using a CLIP-based vision-language model on the subset of Flickr8k dataset with Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) attacks. As a result of the experiment, the performance gap is identified between the clean and perturbed images which confirms the vulnerabilities of the conventional VLM while indicating better robustness consistency with the suggested framework.