Skip to content
Preprint

OmniPack: Unified Token Compression for Efficient Omni-modal Large Language Models

Aug 2026 · 2 citations · 50 references
Computer Science

TL;DR

OmniPack is proposed, a training-free framework that coordinates structural compression before the LLM with task-relevant semantic refinement within the LLM, and consistently achieves the best performance-efficiency trade-off across diverse retention ratios, outperforming all existing methods.

Abstract

Omni-modal large language models (Omni-LLMs) have achieved remarkable performance on audio-visual understanding tasks, but processing long and highly redundant visual and audio token sequences incurs substantial computational overhead, demanding aggressive token compression for efficient deployment. Existing methods often degrade at low token budgets: pre-LLM compression may discard structurally important and globally distributed evidence, whereas inner-LLM compression often underexploits query-conditioned audio-visual collaboration. To address these limitations, we propose OmniPack, a training-free framework that coordinates structural compression before the LLM with task-relevant semantic refinement within the LLM. Before the LLM, OmniPack removes structural redundancy through modality-specific importance, global coverage, and similarity-aware merging. After sufficient multimodal interaction, it further consolidates diverse, task-relevant representations through textual guidance and audio-visual collaboration. Extensive experiments on five benchmarks with three Omni-LLM backbones demonstrate that OmniPack consistently achieves the best performance-efficiency trade-off across diverse retention ratios, outperforming all existing methods. Notably, on Qwen2.5-Omni-7B, OmniPack preserves 98.0% of the original performance while reducing FLOPs to 16.7%, and still retains 92.9% of the original performance with only 6.8% of the original FLOPs.

View source

Similar papers

Preprint Sep 2026

OmniRoute: Mapping Temporal Semantic Evidence to Audio-Visual Token Budgets for Efficient Omnimodal Large Language Models

Omnimodal large language models (Omni-LLMs) encode audio and visual streams into temporally interleaved token sequences for multimodal reasoning. However, processing long audio-visual token sequences incurs substantial prefill costs. Existing compression methods have made progress, but often overlook temporal changes i...

Yu-Chen Deng, Zi-Dang Cai, Fei-Diao Yang et al. · 0 citations
#machine learning Preprint Sep 2026

GleanVID: Complementary Token Selection for Efficient Video Large Language Models

Video Large Language Models (VideoLLMs) have achieved strong video understanding capabilities but incur substantial inference overhead due to the large number of visual tokens. Existing VideoLLM token compression methods largely rely on selection-independent scoring, overlooking cross-frame complementarity and conseque...

Shuo Yang, Chang-Bai Li, Rui Tang et al. · 0 citations
Sep 2026

Resolution as a First-Class Decision: Task-Conditioned Routing for Efficient Multimodal Large Language Models

The inference efficiency of Multimodal Large Language Models (MLLMs) is severely constrained by massive visual token sequences induced by high-resolution inputs, with computational cost scaling quadratically. Existing approaches primarily focus on downstream token compression, while overlooking a fundamental upstream i...

Zhi-Qiang Xia, Yang Li, Xin-Yuan Zhang et al. · 0 citations
#artificial intelligence Preprint Oct 2026

VETO: Video Efficient Token Optimization for Vision Language Models

The key design insight is hierarchical ordering: by first compressing spatial dimensions, VETO drastically reduces the cost of subsequent global temporal matching, bypassing the efficiency wall of single-axis approaches, with an advantage that grows with modern fully-fused attention infrastructure.

Gueter Josmy Faure, Hao-Ping Wang, Min-Hung Chen et al. · 0 citations

VPRune: Efficient Training-free Pre-LLM Visual Token Pruning

Experiments on FastVLM-1.5B across multiple vision-language benchmarks demonstrate that VPRune achieves a favorable accuracy--compression trade-off, with particularly pronounced advantages under aggressive compression, demonstrating its practicality for resource-constrained LVLM deployment.

Guang-Chuan Lv, Dian-Xing Shi, Ding-Jie Fu · 0 citations
Sep 2026

P4Q: Co-designing Token Pruning and Quantization for Vision-Language Model Acceleration

Vision language models have achieved strong performance across a wide range of multimodal applications, yet their substantial computational and memory costs hinder efficient deployment. Visual token pruning and post-training quantization reduce inference overhead along two complementary dimensions, namely sequence leng...

Hai-Zhao Jing, Zhen-Hao Shang, Hao-Kui Zhang et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.