Skip to content

Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddings

Sep 2026 · 0 citations · 29 references
Computer Science

TL;DR

SCSP is a training-free framework that leverages semantic compression for informative token selection in long-context embedding and can be integrated into both zero-shot and fine-tuned models in a plug-and-play manner, consistently improving their performance.

Abstract

Large language models (LLMs) have shown strong potential as training-free text encoders for long-context embeddings. Existing approaches primarily improve information flow under causal attention and typically construct embeddings by uniformly averaging all token representations. However, for long documents, such mean pooling can dilute salient semantic information with abundant redundant or weakly informative content. To this end, we propose SCSP, a training-free framework that leverages semantic compression for informative token selection in long-context embedding. Specifically, SCSP first partitions a document into sentence-aware chunks and appends a semantic compression prompt to each chunk. A prompt-isolated attention mask preserves information flow among document tokens while restricting each prompt to its corresponding local context. We then use the attention patterns elicited by these prompts to estimate token importance, select informative tokens, and aggregate their intermediate-layer representations into the final embedding. Extensive experiments on long-context embedding benchmarks demonstrate that SCSP can be integrated into both zero-shot and fine-tuned models in a plug-and-play manner, consistently improving their performance.

View source

Similar papers

#artificial intelligence Preprint Aug 2026

REIGN: Refurbished Embeddings with Integrated Guidance Networks for Efficient Context-Length Scaling

This work proposes REIGN (Refurbished Embeddings with Integrated Guidance Networks), a contrastively trained bi-encoder that operates on sequences of contextualised chunk embeddings from a frozen Guidance Network rather than on raw tokens for document-to-document retrieval.

Devrim Cavusoglu, Emre Akbas · 0 citations
#artificial intelligence Preprint Oct 2026

Enhancing Long-Video VLM Embeddings with Query-Aware Streaming Latent Reasoning

Query-Aware Streaming Latent Reasoning (QASLR), a post-training framework that accumulates evidence across clips while keeping the embedding size fixed, and support streaming latent aggregation as an effective approach to integrating long-video evidence into fixed-dimensional representations.

Hao-Zhe Chi, Song Jin, Yang Jin et al. · 1 citation
Preprint Sep 2026

Layer-Aware Position Embeddings for Visual Token Pruning in Multimodal Large Language Models

Multimodal large language models (MLLMs) incur substantial computational overhead due to the reliance on hundreds of visual tokens to represent images. While token pruning has emerged as a promising approach to reduce the inference cost of MLLMs, existing methods typically reassign position embeddings to the retained t...

Ya-Hong Wang, Zhang-Kai Ni, Jun-Cheng Wu et al. · 0 citations

Vox-Infinity: Benchmarking the Limits of Long-Context Spoken Language Models

Long-context understanding remains a fundamental challenge for large language models, as excessively long inputs often lead models to forget salient information. This issue is even more pronounced in the speech domain, where audio, as a low-compression modality, requires substantially more embeddings than text to prese...

Xi-Ze Cheng, Wen-Xu Jia, Chenyuhao Wen et al. · 4 citations · ⚡1

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.