Skip to content

SPIDER: Multi-Layer Semantic Token Pruning and Adaptive Sub-Layer Skipping in Multimodal Large Language Models

Sep 2026 · IEEE Transactions on Pattern Analysis and Machine Intelligence · Vol PP · 0 citations · 43 references
Computer Science Medicine

TL;DR

It is proposed that intermediate layer tokens from vision encoders should be considered for effective visual token pruning, as semantic focus shifts across layers, with middle-layer tokens capturing more detailed object-centric information that deeper layers may abstract away.

Abstract

Multimodal Large Language Models face significant efficiency challenges that stem from two distinct yet coupled sources: data redundancy and computational redundancy. While most methods focus on data redundancy by pruning visual tokens from the output of the visual encoder or computing redundancy in LLM decoders using blockwise importance, the finer-grained inter-layer representation shifts and the distribution differences within the layers themselves have not been fully explored. In this work, we comprehensively investigate this dual-level inefficiency. We posit that intermediate layer tokens from vision encoders should be considered for effective visual token pruning, as semantic focus shifts across layers, with middle-layer tokens capturing more detailed object-centric information that deeper layers may abstract away. Furthermore, we reveal the differential contributions of Attention and FFNs across distinct LLM decoder layers. Building upon these discoveries, we propose SPIDER, a training-free framework that integrates multi-layer Semantic visual token PrunIng with an aDaptive sub-layER skipping mechanism. Experimental evaluations demonstrate that SPIDER consistently maintains strong performance across various MLLM architectures and reduction ratios. For instance, on LLaVA-NeXT-7B, SPIDER reduces FLOPs by $80\%$ while maintaining 96$\%$ of the baseline performance.

Read PDF

Similar papers

Conference Open access Sep 2026

Shallow to Deep: Aligning Token Pruning with Stage-wise Roles in LVLMs

STD is proposed, a hierarchical token pruning framework that adapts token selection mechanisms to the functional role of each network stage, and introduces a Stability-Adaptive Trigger in deep layers to execute pruning only during semantically stable phases.

Shuo Zhang, Jin-Tao Tong, Yi-Xiong Zou et al. · 0 citations
#machine learning Preprint Oct 2026

DIPrune: Task-Aware Token Pruning with Dual Importance for Efficient Multimodal Language Models

Recent training-free pruning approaches for Multimodal Large Language Models (MLLMs) effectively cut computational overhead by exploiting visual redundancy or text-vision attention. However, they frequently suffer from semantic degradation due to their task-agnostic design or unreliable attention estimates. Based on ou...

Shuo Yang, Chang-Bai Li, Lin-Lin Yang et al. · 0 citations
#artificial intelligence Preprint Sep 2026

SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models

SinkPruner is proposed, a training-free visual token pruning framework for efficient MLLM inference that follows a coarse-to-fine design with two key modules: a visual sanitizer that filters high-norm redundancies and alleviates attention sink and attention dispersion, and a text-guided pruner that further retains toke...

Shi-Yu Li, Zi-Yuan Hu, Shijia Huang et al. · 2 citations
Preprint Sep 2026

S$^2$Prune: Spatially Structured Visual Token Pruning for Multimodal Large Language Models

S$^2$Prune is proposed, a training-free pruning method that preserves spatial coverage while adapting token density to local image structure and achieves the highest average accuracy among the evaluated training-free pruning methods.

Yuan-Yuan Jia, Shun-Pu Tang, Qian-Qian Yang · 1 citation · ⚡1
#artificial intelligence Preprint Sep 2026

STAR-Pro: Stage-Wise Token Adaptive Reduction with Progressive Refinement for Efficient Large Vision-Language Models

These findings show that effective pruning should preserve broad visual coverage before fusion and progressively refine the retained tokens as cross-modal evidence evolves during fusion, and propose STAR-Pro (STage-Wise Adaptive Token Reduction with Progressive Refinement), a training-free two-stage framework.

Yi-Chen Guo, Ting-Hao Wang, Qi-Zhe Zhang et al. · 1 citation
Sep 2026

Representation Dynamics Reveal Semantic Saliency and Similarity for Visual Token Pruning in MLLMs

Multimodal large language models (MLLMs) incur high inference latency from long visual token sequences. Existing pruning methods commonly use attention maps or output features to estimate token importance or redundancy. Several recent approaches also exploit representation changes, but when and how these changes reflec...

Wei-Xuan Li, Zi-Kun Zhou, Xin-Yi Zhuang et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.