CMSP: a cross-modal similarity preservation method to improve CLIP performance

Abstract

As a fundamental task in multimedia information retrieval, cross-modal image-text retrieval connects heterogeneous visual and linguistic modalities to support bi-directional image and text search by creating semantic correlations. While foundational models represented by CLIP provide high-quality baseline features and impressive zero-shot capabilities, their direct application to specific target domains is frequently hindered by pre-training distribution discrepancy. Current lightweight adaptation techniques are often inadequate for complex cross-modal scenarios, occasionally leading to feature degradation rather than enhancement. To address the issue where implicit structural preservation often fails under complex data distributions, a Cross-modal Similarity Preservation (CMSP) method is proposed. Drawing on the concept of knowledge distillation, this method explicitly constrains the enhanced features via Kullback-Leibler (KL) divergence, ensuring that the resulting feature space maintains similarity distributions consistent with the original CLIP feature space across both intra-modal and cross-modal dimensions, while simultaneously demonstrating greater robustness in retrieval performance. Experimental results on multiple standard datasets verify the effectiveness of CMSP, providing an efficient and lightweight solution for cross-modal image and text retrieval tasks.