Skip to content

ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transport

Sep 2026 · arXiv.org · Vol abs/2609.34899 · 1 citation · 47 references
Computer Science

TL;DR

ColNanoVDR is presented, to the authors' knowledge the first framework to bring this document-free distillation to multi-vector VDR, and it is proved that the resulting alignment cost bounds the MaxSim score difference on every page.

Abstract

Multi-vector retrievers built on vision-language models lead visual document retrieval (VDR), but they run a multi-billion-parameter query encoder on every search. Distilling this encoder into a small student that queries the teacher's existing index would remove the bottleneck. The standard recipe, however, matches the teacher's MaxSim scores and so requires encoding and caching every training page, which can reach terabytes of page tokens. NanoVDR avoids pages entirely by training on the teacher's query embeddings alone, but only for single-vector retrievers. We present ColNanoVDR, to our knowledge the first framework to bring this document-free distillation to multi-vector VDR. Its objective, OTW (Optimal Transport with Learned Weights), aligns the student's query tokens with the teacher's by entropic optimal transport, with a learned weight for each student token, and needs no correspondence between the two tokenizations. We prove that the resulting alignment cost bounds the MaxSim score difference on every page. Distilled from five state-of-the-art teachers, the 149M text-only students retain about 95% of their teachers'NDCG@5 on ViDoRe v1-v3 while encoding queries up to 26x faster. Under identical training, OTW matches score distillation while encoding no page and reading 12.6x less cached teacher data.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

ViSAR: Training-Free Adaptive-$k$ Retrieval for Visual Document Question Answering

ViSAR (Visual Semantic Activation Retrieval), a training-free adaptive-$k$ retrieval method for late-interaction visual document retrieval, is introduced and it is shown that the similarity matrix structure correlates with answer accuracy, suggesting future directions for retrieval quality-aware document understanding.

Adrien Mialland, Marc Plantevit, Julien Gallois et al. · 2 citations
Preprint Oct 2026

EVIE: Evidence-Vector-Informed Embeddings for Visual Document Retrieval

This work introduces EVIE (Evidence-Vector-Informed Embeddings), a family of native visual document retrievers integrating three key innovations: evidence-judged data governance, which uses a multimodal judge to identify answer-bearing positives and filter unreliable negatives, and Hierarchical agglomerative index comp...

Zi-Fei Wang, Wei Wen, Qian Ji et al. · 1 citation
Preprint Sep 2026

MAGIC: Marginal-Guided Compression with Optimal Transport for Efficient Visual Document Retrieval

Recent visual document retrieval (VDR) systems such as ColPali use multi-vector page embeddings, in which patch-level vectors enable fine-grained evidence matching but incur substantial index storage and MaxSim scoring overhead. Post-hoc merging offers a practical route to efficient VDR by reducing this cost without re...

Xu Yuan, Huai-Zhi Liu, Wen-Qi Fan et al. · 0 citations
Book Open access Sep 2026

AdaMerge: Tuning-Free Patch Compression for Multi-Vector Visual Document Retrieval

AdaMerge is proposed, a plug-and-play compression method that detects each document's own cliff via gap analysis on the merge-cosine trajectory, and builds attention-weighted cluster centroids to preserve salient signal.

J.-H. You, Kun Ni · 0 citations
Preprint Sep 2026

Test-Time Adaptation with Query-Dependent Residuals for Visual Document Retrieval

Visual document retrieval (VDR) systems depend on page embeddings computed before deployment, which makes adaptation difficult when encoder parameters or corpus re-encoding are unavailable. Rerankers provide useful relevance signals, but conventional reranking applies them only to selected queries and candidate pages....

Ze-Liang Li, Xiao-Fen Xing, K. Guo et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Jun 3, 2026

MIT researchers teach AI models to interpret charts

The new ChartNet training dataset could improve the accuracy of vision-language models that help analyze business trends or interpret scientific figures.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.