Vision-language models face a fundamental scaling bottleneck: the number of visual tokens grows with both temporal duration and spatial resolution, making long-video understanding expensive for the vision encoder and the language model. Existing methods often compress visual tokens after dense encoding, creating a mism...
Yulong Liu, Xiao-Tian Han, Jun-Yuan Shang et al.· 0 citations
OMP-MoE, a novel training-free compression framework for reducing expert redundancy in MoE-based LLMs by reformulate the pruning problem as a sparse signal reconstruction task solved through Orthogonal Matching Pursuit.
De-Zhi Li, Lu-Jun Li, Qi-Yuan Zhu et al.· 0 citations
Across reasoning, long-context understanding, and agentic tasks, SAS consistently outperforms trainable sparse attention baselines across attention budgets, with especially large gains under tight budgets, demonstrating more effective context ranking for downstream tasks.
Test-time training is formulated as read-write TTT serving and present RW-TTT, which tags each decode step with its owner, version, and READ/WRITE effect, batches only compatible phases, and commits updates only to the owner.
Hierarchical Landmark Sparse Attention is proposed, a chunk-wise sparse attention mechanism that learns chunk selection end-to-end under the language-modeling (LM) loss, enabling long-context LLMs that are both more efficient and more effective on general long-context tasks than their full-attention counterparts.
ReCo (Reward-Coordinated Compression), a step-wise framework in which a lightweight process-reward estimator scores each completed step and drives three components: reward-adaptive KV-cache compression that shrinks the retained cache harder at high-reward steps and less at low-reward ones, and a confidence-based early...
Qi-Yuan Zhu, De-Zhi Li, Pengyu Cheng et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
