Skip to content
Book Open access

Learning Universal Semantic Simplex for Jailbreaking Large Vision-Language Models

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · pp. 3463-3472 · 0 citations

Abstract

Despite the remarkable capabilities of Large Vision-Language Models (LVLMs), they remain vulnerable to jailbreak attacks. The integration of visual modalities further expands the attack surface due to its high-dimensional nature. To ensure the secure development of LVLMs, it is essential to develop more potent jailbreak attacks to probe their safety boundaries. This work focuses on universal attacks as they better reflect real-world scenarios. However, achieving robust universality remains challenging due to the tendency to overfit specific semantics within the training corpus. To bridge this gap and enhance generalization across diverse instructions, we propose Learning Universal Semantic Simplex (LUSS), a novel jailbreak method that shifts the optimization paradigm from point-wise perturbations to continuous adversarial simplices. By incorporating a sharpness-aware penalty and a simplex-based update strategy, LUSS captures a broader range of adversarial semantics within a convex hull to mitigate overfitting. To facilitate more rigorous evaluation, we introduce HarmInstBench, a manually curated benchmark with hierarchical annotations for diverse malicious scenarios. Extensive experiments on multiple LVLMs, including MiniGPT-4, LLaVA-v1.5, and Qwen2.5-VL, demonstrate that LUSS significantly outperforms state-of-the-art baselines, achieving up to a 60.8% relative improvement in Attack Success Rate (ASR), 31.8% in toxicity score, and a 44.7% gain in Malicious Intent Fulfillment Rate (MIFR). Warning: this paper contains harmful text!

Read PDF

Similar papers

Open access Oct 2025

Robust Reasoning Segmentation for Open-World MLLMs without Catastrophic Forgetting

Ensuring trustworthiness and versatility in Open-World Visual Recognition is a core goal of Multimodal Large Language Models (MLLMs). While integrating segmentation is desirable, prevailing methods require invasive fine-tuning, which often leads to catastrophic forgetting of the model’s general-purpose capabilities. Th...

Jia-Zhen Liu, Long Chen · 8 citations
Oct 2026

Flexible Targeted Adversarial Alignment with Frequency-based Visual Focus for Attacking Large Vision-Language Models.

Although Large Vision-Language Models (LVLMs) have shown great capabilities across various downstream tasks, they are proven to be vulnerable to adversarial examples, thus resulting in critical safety issues. Existing adversarial attacks against LVLMs predominantly rely on uniform, image-wise perturbations, while large...

Dai-Zong Liu, Tian-Yao Luo, Junhao Dong et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Jailbreak Connectivity: Towards Diverse, Transferable, and Universal Multimodal Large Language Model Jailbreak

Multimodal large language models (MLLMs) are vulnerable to visual jailbreak attacks that bypass safety safeguards. Existing methods typically optimize isolated adversarial examples, limiting attack diversity, transferability, and generalization. We propose Jailbreak Connectivity (JC), a framework based on the observati...

Xu Zhang, Zi-Qing Hu, Shuo Han et al. · 0 citations
#artificial intelligence Conference Open access Oct 2026

ABench: Adversarily Benchmarking Large Language Models across Taxonomized Values

ABench is a fine-grained, handcrafted benchmark specifically tailored to taxonomized values in Chinese, grounded in a rigorous hierarchical taxonomy comprising 158 derived value subcategories and utilizing question-specific scoring criteria for nuanced assessment.

Li-Ping Yang, Yong-Qi Leng, Jing-Ting Zheng et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.