Skip to content
Book Open access

AdaMerge: Tuning-Free Patch Compression for Multi-Vector Visual Document Retrieval

Sep 2026 · Proceedings of the 35th ACM International Conference on Information and Knowledge Management · 0 citations · 12 references
Computer Science

TL;DR

AdaMerge is proposed, a plug-and-play compression method that detects each document's own cliff via gap analysis on the merge-cosine trajectory, and builds attention-weighted cluster centroids to preserve salient signal.

Abstract

Multi-vector visual document retrieval (VDR) models such as ColPali and ColNomic achieve strong accuracy by representing each document with hundreds to thousands of patch-level embeddings, at substantial storage and latency cost. Existing compression methods either prune unimportant patches or merge similar ones into clusters; the recent state-of-the-art merging method Prune-then-Merge (PtM) consistently outperforms pruning-only baselines at high compression, but requires a per-dataset cluster budget m to be tuned by grid search. We observe that the merge-cosine sequence produced by hierarchical clustering exhibits a sharp cliff separating mergeable redundancy from salient signal, and that the location of this cliff is concentrated in a narrow band across more than 11,000 documents from 14 datasets. This suggests the merge boundary can be detected per document rather than tuned per dataset. Building on this observation, we propose AdaMerge, a plug-and-play compression method that (i) detects each document's own cliff via gap analysis on the merge-cosine trajectory, and (ii) builds attention-weighted cluster centroids to preserve salient signal. On the long-document benchmark ViDoRe-V2 (4 datasets, two backbones), AdaMerge significantly outperforms tuned PtM across the operating range (p < 10-4); on the short-document benchmark ViDoRe-V1 (10 datasets, two backbones), where all merging methods are already near-lossless, AdaMerge matches tuned PtM without any per-dataset tuning. AdaMerge adds only about 10≈ms per document and exposes a single global hyperparameter shared across all datasets and backbones.

Read PDF

Similar papers

#artificial intelligence Preprint Aug 2026

Spatial Matryoshka Training for Multi-Granularity Visual Document Retrieval

It is demonstrated that models trained using ColSNAP maintain near full-resolution retrieval performance under substantial compression and that ColSNAP transfers effectively across multiple late-interaction backbones, and achieves most of its improvements via a lightweight adaptation stage applied to a pre-trained retr...

T. Roy, Arkadeep Acharya, Vishwajeet Kumar et al. · 1 citation
Preprint Sep 2026

Generative Late-Interaction Embeddings For Visual Document Retrieval

Generative Late-Interaction Embeddings (GLIE): k<<N vectors per page learned from the normalized centroids to serve as both a lightweight index and a basis for regenerating the page's full embedding set, with the decoder as its main design surface.

Mohamed Eltahir, Talal Aloushan, Rose Khairoalsendi et al. · 0 citations
Preprint Sep 2026

MAGIC: Marginal-Guided Compression with Optimal Transport for Efficient Visual Document Retrieval

Recent visual document retrieval (VDR) systems such as ColPali use multi-vector page embeddings, in which patch-level vectors enable fine-grained evidence matching but incur substantial index storage and MaxSim scoring overhead. Post-hoc merging offers a practical route to efficient VDR by reducing this cost without re...

Xu Yuan, Huai-Zhi Liu, Wen-Qi Fan et al. · 0 citations
Preprint Oct 2026

EVIE: Evidence-Vector-Informed Embeddings for Visual Document Retrieval

This work introduces EVIE (Evidence-Vector-Informed Embeddings), a family of native visual document retrievers integrating three key innovations: evidence-judged data governance, which uses a multimodal judge to identify answer-bearing positives and filter unreliable negatives, and Hierarchical agglomerative index comp...

Zi-Fei Wang, Wei Wen, Qian Ji et al. · 1 citation
#computer vision Sep 2026

ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transport

ColNanoVDR is presented, to the authors' knowledge the first framework to bring this document-free distillation to multi-vector VDR, and it is proved that the resulting alignment cost bounds the MaxSim score difference on every page.

Zhu Liu, Zi-Yi Wang, Yao Zhang et al. · 1 citation
Preprint Aug 2026

Learning Sample-wise Rank-aware Interpolation Weights for Composed Visual Data Retrieval

This work revisits the efficacy of simple linear interpolation within an embedding space, and introduces SRAIN, the first framework that dynamically predicts query-specific interpolation weights, and achieves the best in composed video retrieval and matches the current state of the art in composed image retrieval.

Boseung Jeong, T. Park, Donghyeon Kwon et al. · 1 citation

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.