PWFAR: Patch–Word Fine-Grained Alignment for Long-Text Image–Text Retrieval

Abstract

The main challenge in long-text image–text retrieval lies in the fact that some existing vision–language models mainly rely on global image and text features for matching. Without explicit local alignment constraints, these models may struggle to fully capture complex semantic information in long textual descriptions, such as local objects, fine-grained attributes, and spatial relationships. To address the insufficient modeling of local details in global semantic matching, this paper proposes a patch–word fine-grained alignment retrieval model, named PWFAR. Built upon the Long-CLIP framework, the proposed method introduces an explicit fine-grained alignment mechanism between image patches and text tokens. By using textual words to guide the matching of local image regions, PWFAR enhances the model’s ability to capture local semantic correspondences. Specifically, PWFAR consists of three complementary training objectives: long-text global contrastive learning, short-text compact semantic supervision, and patch–word fine-grained alignment. These objectives are jointly optimized to constrain overall semantic consistency, core semantic stability, and local detail matching relationships. The model is trained on the 888k subset of the ShareGPT4V dataset and evaluated on the COCO2017 and Urban1k datasets. Experimental results show that PWFAR achieves competitive performance across different backbone networks and retrieval directions. Its clearest and most consistent gains are observed on the Urban1k long-text retrieval task, where it outperforms Long-CLIP and Long-CLIP-888k in both retrieval directions. These results indicate that the proposed fine-grained alignment mechanism can effectively improve the model’s ability to capture local semantic relationships in long texts, thereby verifying the effectiveness of PWFAR for long-text image–text retrieval.