Jul 2026· International Conference on Machine Vision, Automatic Identification and Detection· Vol 14261, pp. 1426114 - 1426114-6· 0 citations· 9 references
Engineering
TL;DR
Empirical evaluations across a diverse suite of multimodal document retrieval benchmarks reveal that DART achieves consistent and significant gains in ranking precision, and this dynamic refinement process introduces minimal computational latency, offering a highly efficient, plug-and-play solution for adaptive document retrieval.
Abstract
While Multimodal Large Language Models (MLLMs) exhibit strong capabilities in document retrieval, their zero-shot performance is frequently bottlenecked by a static inference paradigm. Once deployed, these frozen models struggle to dynamically adjust their representation spaces to accommodate highly diverse query semantics and intricate visual layouts. To overcome this inherent rigidity, we introduce Dynamic Adapter Refinement at Test-time (DART), a novel inference-stage adaptation strategy. Rather than fine-tuning the massive backbone or applying destructive global feature projections, DART injects a lightweight, residual Adapter network to re-rank the initial top-K candidates. Our method extracts soft pseudo-labels directly from the model's initial retrieval confidence to drive a self-supervised contrastive learning objective. Crucially, to prevent catastrophic representation drift and preserve the rich vision-language alignment acquired during pre-training, DART constrains the Adapter's parameter updates using a targeted 𝐿𝐿2regularization penalty. Empirical evaluations across a diverse suite of multimodal document retrieval benchmarks reveal that DART achieves consistent and significant gains in ranking precision. Furthermore, this dynamic refinement process introduces minimal computational latency, offering a highly efficient, plug-and-play solution for adaptive document retrieval.
Large Language Models (LLMs) have emerged as powerful assets for recommender systems. However, deploying them as generative recommenders or zero-shot rankers at web-scale remains bottlenecked by prohibitive computational overhead and grounding challenges. In this paper, we revitalize the classic, highly efficient two-tower retrieval architecture by adapting LLMs as semantic representation backbones rather than generative engines. We introduce an LLM-native two-tower framework engineered for high-throughput, large-scale retrieval. Our architecture introduces several key innovations: a shared LLM encoder for joint user-item modeling, End-Of-Sentence (EOS) token pooling for compact sequence embedding, cross-dataset transfer learning, knowledge distillation from powerful cross-encoder teachers, and latent reasoning within the user tower. Extensive evaluation across three public benchmarks demonstrates that cross-encoder architecture outperforms current state-of-the-art (SoTA) models, while the efficient two-tower student achieves SoTA-comparable retrieval performance. Furthermore, experiments on internal large-scale production systems yield substantial topline retrieval improvements along with high resilience to model staleness and superior data scaling. Our findings demonstrate that when augmented with modern representation learning, the traditional two-tower paradigm remains an exceptionally competitive and practical solution for industrial retrieval systems.
The Adaptive Multi-Stage Vector Retrieval (AMSVR) framework is proposed, prioritising weighted, drift-resistant composition over uniform fusion, and offers tailored configurations: AMSVR-Scientific (dense + tuned hybrid) peaks at NDCG@10 = 0.7570 on SciFact, while AMSVR-Full (seven stages) targets broader, noisier corpora where Recall@100 matters most.
Samsudeen Alabi Bankole, Yakub Kayode Saheed· NLP & Big Data· 0 citations
Video Corpus Moment Retrieval (VCMR) is pivotal to multimodal understanding. However, existing methods rely heavily on large-scale annotated data, which limits their generalization and scalability. To address this issue, we propose a training-free VCMR framework, termed Synergistic Collaboration and Adaptive Calibration (SCAC), enabling effective semantic parsing and precise temporal localization without parameter updates. SCAC introduces a Query Event Chain Generation module that leverages large language models to transform complex textual queries into structured event chains, while a Video Event Chain Generation module represents videos as semantically coherent event chains through subtitle segmentation and keyframe aggregation. Built on these structured representations, SCAC performs Event-Chain-Based Cross-Modal Retrieval with mean-variance joint scoring to suppress local mismatches and reinforce global consistency. During localization, a Synergy-Calibration Mechanism dynamically refines temporal boundaries via profit-setback feedback. Extensive experiments show that SCAC achieves comparable or superior results to supervised counterparts under training-free conditions, demonstrating strong cross-modal generalization and adaptive capability. The code of our method is available at https://github.com/cyanlll/SCAC
Jialong Zhao, Huafeng Li, Yafei Zhang et al.· IEEE Transactions on Image P...· 0 citations
Document Visual Question Answering (DocVQA) often leverages Retrieval-Augmented Generation (RAG), where late-interaction encoders are commonly used to identify document pages relevant to a user query, before answer generation by a Large Vision-Language Model (LVLM). Existing approaches typically retrieve a fixed top-$k$ number of pages regardless of query complexity, which increases LVLM latency and may degrade answer accuracy. We introduce ViSAR (Visual Semantic Activation Retrieval), a training-free adaptive-$k$ retrieval method for late-interaction visual document retrieval. ViSAR operates directly in the embedding space to construct a query-conditioned page-level similarity matrix that highlights query-relevant semantics and dynamically determines the number of pages to retrieve. Across multiple encoders and LVLMs, ViSAR retrieves compact, query-adapted page sets that reduce RAG latency by up to 58.7\%, while maintaining or improving answer accuracy compared with fixed top-$k$ and adaptive retrieval heuristics. Furthermore, we show that the similarity matrix structure correlates with answer accuracy, suggesting future directions for retrieval quality-aware document understanding.
Adrien Mialland, Marc Plantevit, Julien Gallois et al.· 0 citations
Retrieval-augmented generation (RAG) has emerged as a promising solution to enhance the reliability of large language models (LLMs) with external knowledge. Existing RAG methods operate in explicit representation spaces: in-context methods inject knowledge through text tokens in the input, while parametric methods like Parametric RAG encode documents into model parameters. Although effective, these approaches face inherent limitations. In-context injection suffers from quadratic computational complexity with context length and degraded performance in complex reasoning tasks. Parametric injection, while reducing inference costs, requires substantial storage overhead and computationally expensive offline preprocessing. More fundamentally, both paradigms rely on explicit discrete representations tokens or parameters that may introduce information bottlenecks and hinder seamless knowledge integration. To address these challenges, we introduce Latent RAG, a novel paradigm that performs knowledge injection entirely within the continuous latent space. Our approach encodes documents into ultra-compact latent representations through an offline compression phase, and directly fuses them with the LLM's hidden states via a learned injection mechanism during inference. By operating in the semantic latent space rather than explicit token or parameter spaces, Latent RAG enables more natural knowledge integration while achieving 9,200X storage reduction compared to Parametric RAG. Experimental results on multiple RAG benchmarks demonstrate that Latent RAG substantially enhances both effectiveness and efficiency. Furthermore, it can be seamlessly combined with existing in-context and parametric methods to achieve even better performance.
Shuran Zhou, Junan Chen, Rui Ling et al.· Annual International ACM SIG...· 0 citations
Composed video retrieval (CoVR) searches a gallery for the target video that realizes a natural-language modification of a source clip. However, at gallery scale, this creates a fundamental tension: compact embeddings enable efficient, reusable search but can miss the transient actions, state changes, and subtle constraints that demand fine-grained video reasoning, whereas applying large multimodal models uniformly sacrifices scalability. To address these limitations, we propose that frozen foundation models should instead occupy complementary roles, with inference depth adapted to query difficulty. Based on this premise, we introduce \methodname{}, a framework for training-free \methodexpansion{}. Specifically, a composed-query embedding first searches reusable video-only gallery representations; uncertain queries undergo bounded reranking and candidate expansion; ambiguous edits trigger target-description generation; and only close leading candidates reach multimodal verification. To support these roles, frame selection, spatial resolution, and time cues are adapted to each stage. Across complete target-gallery evaluations, our method reaches state-of-the-art performance among training-free approaches, with 89.55 and 93.43 R@1 on Dense-WebVid-CoVR and CoVR-R, respectively (with more than +35\% and +25\% absolute margins to the closest counterpart). These results show that adaptively orchestrating foundation-model capabilities can combine scalable retrieval with fine-grained reasoning without task-specific training. The source code and all relevant guidelines are available on https://github.com/demidovd98/CoVRAGE.
Dmitry Demidov, Muhammad Zaigham Zaheer, Omkar Thawakar et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
