Vision-language retrieval with CLIP-style dual encoders achieves strong cross-modal performance, yet practical accuracy often hinges on localized semantic distinctions where top-ranked near misses differ from the true match by a single critical detail. Hard-sample mining can select confusable candidates but cannot construct corrected counterparts; synthetic augmentation can generate novel samples but, without conditioning on actual model failures, targets irrelevant dimensions of hardness. We observe that a top-ranked false positive is a counterfactual scaffold---sharing most of the query's semantics while differing in a localized failure-causing residual. Minimally correcting this residual yields a hard positive of the ground truth in the same modality; the corrected and unedited versions form a hard negative pair that straddles the decision boundary, producing complementary pull--push supervision. We introduce RePair, guided by three principles---Validity, Minimality, and Locality---which mines false positives bidirectionally, applies LLM-guided counterfactual editing, and trains with a local hard-pair contrastive objective. On Flickr30K and COCO30K, RePair outperforms controlled augmentation baselines with only 107K synthetic samples---26\%--75\% fewer than comparable methods---confirming failure-conditioned repair is more data-efficient than error-agnostic augmentation.
Siyi Liu, Xiao-Rong Zhu, En-Jun Du et al.· 0 citations
Large vision-language models used as listwise rerankers must jointly process visual tokens from tens of candidates per query, making token pruning essential for practical deployment. Existing pruning methods retain tokens by attention saliency, yet we show that saliency is systematically misaligned with ranking contribution: visually prominent tokens often capture order-neutral patterns shared across candidates. This mismatch is layer-dependent: saliency becomes informative only where attention is concentrated, and normalized attention entropy diagnoses the reliability shift (Pearson r=0.87). We propose RaDiCal (Rank-Discriminative Calibration), a training-free framework that uses normalized attention entropy to decide when saliency can be trusted, fusing it with an attention-free rank-discriminative prior and selecting pruning layers from the same trust landscape. Across three retrieval benchmarks and multiple VLM architectures, RaDiCal matches Dense MRR@10 on Flickr30K and surpasses it on MSCOCO at a 20% token budget, ranks first among all pruning methods on FashionIQ, and holds within 1.2 pp on Flickr30K and MSCOCO at 10% retention. It cuts FLOPs by 39--45% and delivers 1.28--1.45$\times$ measured speedups across two VLM architectures without dataset-specific retuning.
Siyi Liu, Han-Jun Yang, Chen-Chen Zhang et al.· 0 citations
Multi-objective ranking serves as the backbone of industrial information retrieval, requiring a holistic assessment of documents across dimensions such as Relevance, Authority, and Recency. The prevailing industry paradigm relies on ensembles of specialized BERT-based models, which are costly to maintain and fundamentally limited in expressiveness by the modest model size of discriminative encoders. While Large Language Models (LLMs) offer a path to unified reasoning, their auto-regressive latency is prohibitive for real-time serving. In this paper, we propose UniRank, a Homogeneous Scaling framework that consolidates multiple ranking objectives into a single, unified generative model. To reconcile the conflict between deep reasoning and inference speed, we introduce two paradigmatic innovations: (1) Parallel Prediction via Task-Specific Anchors: We replace slow sequential decoding with a parallel mechanism that predicts all objective scores simultaneously in a single forward pass, achieving O(1) inference complexity. (2) Reinforcement Learning Optimization with Posterior CoT: We propose a novel training framework that synergizes Posterior Chain-of-Thought with GRPO. By forcing the model to generate rationales after scoring and aligning them via policy optimization, we effectively distill complex reasoning capabilities into the anchor tokens without incurring inference-time overhead. Extensive experiments on the MS MARCO benchmark and a massive industrial dataset from Tencent demonstrate that UniRank significantly outperforms BERT-Large and matches the performance of larger LLMs, all while maintaining strictly low latency. Online A/B testing further confirms its practical value, showing that UniRank simplifies the deployment pipeline while delivering a significant lift in core business metrics.
Dezhi Ye, Junwei Hu, Xiaoyang Chen et al.· Proceedings of the 32nd ACM...· 0 citations
Real-world image search queries are multimodal and compositional: ``find this shirt in pink''specifies an entity to retain, an attribute to modify, and context to ignore. Yet existing re-rankers either compress such multifaceted relevance into an opaque embedding or rely on free-form chain-of-thought that easily omits or hallucinates fine-grained constraints. Drawing on rubric- and checklist-based evaluation from NLP, we recast multimodal image re-ranking as a semantic constraint satisfaction problem and propose EviRank, which parses any query - text-only, image-only, or composed - into a unified evidence package: typed criteria across six semantic slots (e.g., entities, attributes, relations), each labelled required, forbidden, or ignorable. Re-ranking then reduces to evidence-conditioned verification, combining deterministic rubric scoring and evidence-grounded listwise comparison in a single training-free procedure. The explicit evidence can further serve as structured supervision for optionally distilling a lightweight student. Across five benchmarks spanning text-to-image, image-to-image, and composed image retrieval, EviRank achieves state-of-the-art performance, and the distilled student preserves over 90% of the teacher's capability at substantially lower cost.
Enjun Du, Siyi Liu, Zi-Rong Chen et al.· 1 citation
These findings suggest that, within the pointwise scoring paradigm, routing continuous relevance semantics through discrete text constrains ranking signal resolution reveals a bottleneck that is stable and difficult to overcome under current standard methods, rather than an easily resolvable training bias.
Xiaoyang Chen, Jie Liu, Haijin Liang et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
