It is observed that attention scores from both vision and text tokens peak at modality separator tokens, suggesting that these separators bridge the two modalities and proposes SepPrune, an efficient, training-free, plug-and-play pruning method that uses the separator token as a unified query to rank and select informative vision tokens.
Abstract
Recent multimodal large language models (MLLMs), such as Qwen2.5-VL and InternVL3, generate large numbers of vision tokens for high-resolution inputs, leading to substantial computational cost. Existing vision token pruning methods either depend on cross-modal attention and cannot prune before the prefill stage, or rely on diversity estimation with high computational overhead. We observe that attention scores from both vision and text tokens peak at modality separator tokens, suggesting that these separators bridge the two modalities. Based on this observation, we propose SepPrune, an efficient, training-free, plug-and-play pruning method that uses the separator token as a unified query to rank and select informative vision tokens. SepPrune reuses the LLM's built-in projection parameters and requires no architectural changes. Experiments on Qwen2.5-VL-7B show that SepPrune achieves state-of-the-art performance, retaining 96.3% of the original accuracy while removing 80.2% of vision tokens.
This work proposes Single-Forward Pruner (SFPruner), a structural reformulation of visual token pruning that embeds redundancy control directly into the scoring space, bypassing the need for iterative combinatorial optimization and achieves redundancy-aware importance selection in a single forward pass.
J. Song, Woohyeong Kim, Kyeongbo Kong· arXiv.org· 0 citations
Compared to prior CLIP-enhancement methods, MLLMCLIP achieves state-of-the-art compositional accuracy while delivering consistent gains on standard zero-shot classification and image-text retrieval, showing that feature-level distillation strengthens both compositional and general vision-language representation capability.
Jongsuk Kim, Qiyu Wu, Zhuoyuan Mao et al.· 0 citations
Existing research on Efficient Multimodal Large Language Models (EMLLMs) primarily focuses on reducing the number of visual tokens in the prefilling stage, which is tailored to short-answer inference scenario. However, in more complex multimodal reasoning tasks, models are often required to generate lengthy intermediate reasoning rationales, while repeatedly revisiting prefilled contexts to verify and revise reasoning paths. As the generation length increases, the cumulative overhead of the decoding stage rises surpasses that of pruned prefilling, to become the dominant cost source for end-to-end inference. Investigating decoding-time attention behaviors, we observe two phenomena on textual and visual side related to selective and effective memory retention. Based on these observations, we propose MMSep, a training-free multimodal separator localization and compression framework that improves efficiency in both prefilling and decoding. MMSep (i) localizes visual anchors/separators during prefilling via question-guided attention and a spatial–similarity constraint, and (ii) performs structured KV-cache compression during decoding by retaining textual separators as long-range context and enabling separator-triggered, on-demand visual recall. Experiments on four MLLM backbones across long-generation and standard reasoning benchmarks demonstrate that MMSep consistently reduces latency while maintaining competitive generation quality and reasoning accuracy. Our code is available at https://github.com/MeinhardMark/MMSep.
Mingjie Ma, Yichao Ma, Jiannan Cao et al.· Proceedings of the 32nd ACM...· 0 citations
Trend-aware Pruning is proposed, a novel framework that elevates pruning from a local snapshot decision to a temporal trajectory modeling problem, and enables a dynamic rectification mechanism that selectively reactivates "late-blooming" tokens, those initially undervalued but exhibiting rising semantic importance, thereby preventing the loss of critical visual cues.
Jie Ma, Zhike Qiu, Jie Gao et al.· arXiv.org· 0 citations
This paper argues that the key lies in designing an appropriate search-state representation that connects the internal knowledge of LLMs with the structural requirements and constraints of visual-token pruning, and proposes AutoPrune, a training-free framework for LLM-driven visual-token pruning policy design.
Zhen Liu, Wenli Huang, Wei Song et al.· 0 citations
A retraining-free VLM pruning framework called PORTA is introduced that derives a task- and modality-agnostic importance formulation based on activation variation, estimated from generic calibration data, which reliably captures feature-level representation utility across modalities.
Minseok Kang, Hyunwoo J. Kim, Chanyoung Kim et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
