On Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspaces

Dual Adversarial Fine-tuning for Enhancing Robustness of Large Vision Language Model

While Large Vision-Language Models (LVLMs), represented by LLaVA and GPT-4V, have demonstrated remarkable capabilities, their visual inputs remain vulnerable to adversarial attacks, posing significant security risks. Existing defense methods predominantly target single-task scenarios (e.g., zero-shot classification) and consequently lack generalizability across various multimodal tasks. To address this limitation, we propose a dual adversarial fine-tuning framework that jointly optimizes visual and semantic supervision signals from two modalities, enhancing model robustness while generalizing across multiple downstream tasks. The proposed framework comprises two core components, i.e., $\textbf{Visual}$ supervision branch and $\textbf{Semantic}$ supervision branch. The former branch leverages features from clean images, extracted via a frozen original vision encoder, to guide adversarial robustness while the latter incorporates caption-image alignment as a contextual signal to preserve semantic coherence under attack. Moreover, our method achieves cross-task robustness by simply replacing the CLIP vision encoder in the original model, with no need of separate task-specific retraining or architecture modifications.Extensive experiments demonstrate that our approach outperforms the state-of-the-art method in adversarial robustness evaluation across zero-shot classification, image captioning, and visual question answering (VQA) tasks.

Multi-layer Adversarial Robustness Analysis of Neural Networks: Visual and Metric-based Approaches

Adversarial examples are purposely crafted to fool Artificial Neural Networks (ANNs) into producing erroneous outputs. Understanding what makes these models vulnerable to such attacks is an open research topic and constitutes a first step toward devising defenses. Focusing on ANNs designed for image classification, we propose a method to analyze these models from an adversarial robustness perspective. The approach is based on the t-distributed Stochastic Neighbor Embedding (t-SNE) technique and allows the visual inspection of the representations learned by an ANN throughout its different layers. Additionally, we propose the Embedding Overlap Robustness RatiO (EORRO) metric to identify potential weak spots in the network by quantifying differences between clean and perturbed images on the t-SNE maps. We validated our approach by studying two ANNs, one designed by humans and another by NeuroEvolution, on CIFAR-10 and CIFAR-100. Our results indicate that differences between clean and perturbed images start to emerge early on in the models, namely in the feature extraction layers of the networks. Guided by the proposed metric, we implemented an ablation and fine-tuning strategy that successfully boosted the robustness of the ANNs against a variant of the Auto-PGD attack under different threat models. More specifically, the post-attack accuracy of the neuroevolved ANN on CIFAR-10 changed from 0% to at least 63.5% in \(L_{2}\) and 27.2% in \(L_{\infty}\) .

Generalization Analysis for Adversarial Vision Transformers

Adversarial Robustness in Large Vision-Language Models

The recent progress in the vision-language model (VLM) research made it one of the key aspects of artificial intelligence due to joint vision and language processing capabilities required for such tasks as image retrieval, visual question answering, autonomous systems and medical image analysis. However, despite the excellent results obtained with such systems, they still appear to be extremely vulnerable to adversarial perturbations which result in a significant decline in prediction accuracy and reliability due to even subtle input transformations. The purpose of this paper is to provide a systematic method for analyzing the adversarial robustness of vision-language models under both clean and perturbed conditions. The suggested framework is based on Semantic Counterfactual Augmentation (SCA) and Curriculum Contrastive Adversarial Training (CCAT) and aims to maintain semantic consistency and increase the robustness of the model respectively. The framework is tested using a CLIP-based vision-language model on the subset of Flickr8k dataset with Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) attacks. As a result of the experiment, the performance gap is identified between the clean and perturbed images which confirms the vulnerabilities of the conventional VLM while indicating better robustness consistency with the suggested framework.

Revisiting Adversarial Robustness in Large-Scale 3-D Vision–Language Models

Pre-trained 3D vision-language models have demonstrated strong semantic generalization and robustness to distribution shifts. However, the implications of semantic robustness for geometric stability remain unclear. This study revisits the adversarial robustness of 3D vision-language models when confronted with adversarial point clouds. Focusing on zero-shot classification, this study demonstrates that these models exhibit heightened sensitivity to small coordinate perturbations. The behavior of adversarial perturbations is further analyzed under widely used point-cloud preprocessing mechanisms, revealing that naive filtering or reconstruction mainly suppresses irregular perturbations produced by vanilla gradient-based attacks and provides limited protection against stronger attack methods. To this end, a refined adversarial objective is introduced with two complementary priors that encourage adversarial point clouds to remain smooth and geometrically plausible: a statistical prior that regularizes the sampling distribution, and a geometric prior that promotes consistency with a plausible object-surface manifold. These findings highlight the need for a more rigorous security evaluation of 3D vision-language models.

Adversarial Purification by Consistency-aware Latent Space Optimization on Data Manifolds.

Deep neural networks (DNNs) are vulnerable to adversarial samples crafted by adding imperceptible perturbations to clean data, potentially leading to incorrect and dangerous predictions. Adversarial purification has been an effective means to improve DNNs robustness by removing these perturbations before feeding the data into the model. However, it faces significant challenges in preserving key structural and semantic information of data, as the imperceptible nature of adversarial perturbations makes it hard to avoid over-correcting, which can destroy important information and degrade model performance. In this paper, we break away from traditional adversarial purification methods by focusing on the clean data manifold. To this end, we reveal that samples generated by a well-trained generative model are close to clean ones but far from adversarial ones. Leveraging this insight, we propose Consistency Model-based Adversarial Purification (CMAP), which optimizes vectors within the latent space of a pre-trained consistency model to generate samples for restoring clean data. Specifically, 1) we propose a Perceptual consistency restoration mechanism by minimizing the discrepancy between generated samples and input samples in both pixel and perceptual spaces. 2) To maintain the optimized latent vectors within the valid data manifold, we introduce a Latent distribution consistency constraint strategy to align generated samples with the clean data distribution. 3) We also apply a Latent vector consistency prediction scheme via an ensemble approach to enhance prediction reliability. (CMAP) fundamentally addresses adversarial perturbations at their source, providing a robust purification. Extensive experiments on CIFAR-10 and ImageNet-100 show that our (CMAP) significantly enhances robustness against strong adversarial attacks while preserving high natural accuracy.