Large vision-language models (LVLMs) achieve strong multimodal understanding, but the hundreds to thousands of visual tokens they process impose substantial computational overhead, motivating training-free visual token pruning. In this work, we conduct two complementary analyses of visual token pruning. First, we measure the feature-space coverage of tokens retained before cross-modal fusion and find that aggressive pruning discards substantial visual information. Second, we track text-to-visual attention across decoder layers and find that the visual tokens considered important change substantially with depth, making one-shot pruning decisions unreliable. Together, these findings show that effective pruning should preserve broad visual coverage before fusion and progressively refine the retained tokens as cross-modal evidence evolves during fusion. We therefore propose STAR-Pro (STage-Wise Adaptive Token Reduction with Progressive Refinement), a training-free two-stage framework. Its Adaptive Stage applies pivoted QR to construct an over-budget feature-coverage candidate pool, while its Progressive Stage uses evolving text-to-visual attention at selected decoder layers to prune a nested survivor set under a target layer-average token budget. Extensive experiments across seven LVLMs spanning multiple architectures and 18 image and video benchmarks demonstrate the effectiveness of STAR-Pro under aggressive pruning. On LLaVA-Video-7B, STAR-Pro reduces visual tokens by 90.5%, retains 92.7% of baseline performance, and achieves a $2.24\times$ measured inference speedup. Code is available at https://github.com/EasonAI-5589/starpro.
Yi-Chen Guo, Tinghao Wang, Qizhe Zhang et al.· 0 citations
In vision-language models (VLMs), visual tokens usually bear a significant amount of computational overhead despite sparsity of information in them when compared to text tokens. To address this, most existing methods learn a network to prune redundant visual tokens using certain training data. Differently, we propose a text-guided training-free token optimization mechanism dubbed SparseVLM without fine-tuning costs. Given that visual tokens complement text tokens in VLM's linguistic reasoning, we select relevant text tokens to rate the significance of visual tokens using self-attention matrices and, then, prune visual tokens using the proposed strategy to maximize sparsity while retaining information. In particular, we introduce a rank-based strategy to adaptively determine the sparsification ratio for each layer, alongside a token recycling method that compresses pruned tokens into more compact representations. However, the Attention Gravity and Attention Sink phenomena undermine the sparsification reference of SparseVLM: text-visual alignment. To address this, we introduce improved text-visual attention patterns to enhance the fidelity of query-aware vision token selection. First, we correct the Attention Gravity effect, where positional encodings induce disproportionately high attention to vision tokens positioned near textual tokens. We further mitigate the Attention Sink by selecting priority attention heads that suppress the excessive attention absorbed by semantically irrelevant vision tokens. The two parts refine the text-visual alignment from both the token and head perspectives. Finally, for video understanding tasks, we propose Temporal-Balanced Sparsification, complementing spatial relevance with temporal diversity to reduce cross-frame redundancy. Extensive experiments on 23 widely used image and video understanding tasks across 7 vision-language models demonstrate the effectiveness of our SparseVLM+ framework. For example, equipping LLaVA with SparseVLM+ yields a 57.1% reduction in FLOPs and a 38.4% decrease in CUDA latency, while preserving 99.6% of its original accuracy on average. Our code is available at https://github.com/Gumpest/SparseVLMs.
Yuan Zhang, Junpeng Ma, Qizhe Zhang et al.· IEEE Transactions on Pattern...· 2 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
