Skip to content
Conference Open access

Auxiliary text-guided image restoration for image-text matching

Sep 2026 · International Conference on Image Processing and Pattern Recognition · Vol 14388, pp. 1438803 - 1438803-6 · 0 citations · 18 references
Engineering

TL;DR

A new ITM framework that improves the model's discriminative performance by focusing on localized core attributes and has a better robustness in handling highly similar hard negatives, which provides a new way to address cross-modal hard sample discrimination.

Abstract

Image-Text Matching (ITM) aims to establish deep semantic associations between visual content and textual descriptions. Existing methods usually have discrimination issues because of fine-grained semantic deviations, so it's hard to capture the complex correspondences between cross-modal entries. Only relying on alignment loss a lot results in a lot of topranked false positives when dealing with such subtle semantic discrepancies. To resolve this issue, we introduce a new ITM framework that improves the model's discriminative performance by focusing on localized core attributes. Specifically, introducing a Semantic Augmentation Strategy (SAS) with a Text-guided Image Restoration (TIR) task, while forcing the model to extract color semantics from grayscale images, our approach effectively creates deep semantic coupling across modalities. Optimized via a multi-loss collaborative framework, AT-JLIM significantly expands the inter-class distance between positive and negative samples in the feature space. Extensive experiments on benchmark datasets, like MSCOC and Flickr30K, show that the proposed method does a lot better in fine-grained matching performance. It has a better robustness in handling highly similar hard negatives, which provides a new way to address cross-modal hard sample discrimination.

Read PDF

Similar papers

Conference 2026

Dual Associations Semantic Enhancement for Image-Text Matching

Experimental results on two publicly available datasets demonstrate that the pro-posed DASE model achieves significant performance improvements in image-text matching tasks compared to baseline methods, validating its effectiveness and superiority.

Xinlin Zhao · 0 citations
Sep 2026

MODdapter: Spatially-aware text embeddings for zero-shot semantic segmentation.

MODdapter extracts visual semantic cues from coarse segmentation results and integrates them as supplementary textual data, allowing for a more accurate projection of text features into the feature space and enhancing fine-grained recognition of unseen objects.

Jia-Xiang Fang, Shi-Qiang Ma, Jing Wang et al. · 0 citations
Sep 2026

BPE-Level Visual-Textual Alignment for Multi-Scene Text Retrieval

Scene Text Retrieval (STR) aims to search images containing a given textual query within large-scale image collections. However, existing approaches are fundamentally constrained in two ways: 1) they are evaluated on narrow benchmarks that focus primarily on natural scenes; and 2) they rely on either error-prone multi-...

Tong-Kun Guan, Yu-Tong Cai, Hao-Cheng Wang et al. · 0 citations
Conference Aug 2026

MaskFlow: Attention-Guided Localized Editing for Rectified Flow Models

Text-guided image editing using rectified flow models such as Multimodal Diffusion Transformer (DiT) has demonstrated impressive generation quality. However, existing methods apply edits globally, inevitably modifying background regions unrelated to the intended semantic change. Our key observation is that the Text-to-...

Trong-Tai Dam Vu, Vinh-Tiep Nguyen · 0 citations
Open access 2026

Semantic-Aware Consistency Rectification for Cross-Modal Person Retrieval

This approach introduces a Paired-Guided Consistency Rectification (PCR) module that utilizes strictly paired data as reliable anchors to dynamically calibrate the confidence of unpaired samples, effectively suppressing noise from inconsistent associations under mild-to-moderate corruptions.

Zi-Xuan Zhang, Lu-Ming Xiao · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.