From Patching to Pruning Visual Computation in Vision Language Models
Vision language models (VLMs) incur substantial inference cost because every visual token is processed by the attention and MLP projections of every decoder layer, even when token-specific visual computation is unnecessary at many depths. We introduce Patch-to-Prune (P2P), inspired by Mechanistic Interpretability, a tr...
Rahul Chowdhury, Timothy Rupprecht, Xuan Shen et al.
· 0 citations