MiERDCIR: Intent-Diverse Synthetic Rewriting for Generalizable Composed Image Retrieval; with an Analysis of Instance-aware Soft Contrastive Learning

Abstract

Composed Image Retrieval (CIR) asks a system to retrieve a target image from a reference image and a natural-language modification. Although large vision-language models provide strong global image-text representations, CIR remains difficult because training data often contains verbose or weakly pragmatic modification texts, while fine-grained instance-level grounding is expensive to supervise directly. This thesis therefore studies CIR from two complementary directions. The primary contribution is MiERDCIR, an intent-diverse synthetic rewriting discipline that rewrites MTCIR modification texts into more pragmatic, retrieval-oriented instructions while keeping the image pairs fixed. The modeling component is an exploration of ScheiCIR, an instance-aware soft contrastive variant that uses noun phrases as weak semantic proxies for visual instances and adds an auxiliary instance-level objective to a global CIR retrieval model. Evaluation on MTCIR, MiERD-MTCIR, FashionIQ, and CIRR shows that the modification-text distribution has a larger effect than the tested instance-level objective. MiERD-MTCIR-trained runs substantially outperform models trained on original MTCIR or LaSCo and transfer strongly to CIRR, reaching up to 29.83 full-gallery Recall@1 and 66.92 Recall subset@1. A diagnostic human rating study, however, shows that the rewritten instructions are not uniformly more faithful: they are much more natural, but they also introduce risks of over-compression, omission, and intent drift. The ScheiCIR instance-aware branch is best interpreted as a diagnostic probe of weak noun-phrase-level supervision. Its loss saturates rapidly and produces only small differences among MiERD-MTCIR-family runs, suggesting that the current auxiliary task is too easy or redundant with the global retrieval objective.