Aggregating Neighbor Embedding Projection and Rank-Based Manifold Learning for Image Retrieval

Abstract

Content-based image retrieval (CBIR) has evolved significantly with the advent of deep learning models, yet effectively ranking similar images remains a challenging task, particularly in high-dimensional feature spaces where pairwise distance measures often fail to capture complex contextual relationships and the semantic gap between visual features and high-level concepts persists. In this scenario, manifold learning and rank-based refinement methods have emerged as complementary strategies, respectively improving feature representations and exploiting contextual information embedded in ranked lists, such as neighborhood relationships among images. However, combining projection-based and rank-based strategies to exploit their complementary properties remains a challenging research problem. To address this, this paper proposes a framework that combines neighbor embedding projections with rank-based manifold learning through rank aggregation. Specifically, Uniform Manifold Approximation and Projection (UMAP) is used to generate alternative low-dimensional feature representations, while ranked lists obtained from UMAP projections and rank-based re-ranking methods are combined using the Borda Count aggregation strategy. The experimental evaluation was conducted on several public datasets using deep learning features extracted from ResNet152, Swin Transformer, and DINOv2 models. The results show that the proposed approach can improve retrieval effectiveness in several scenarios, particularly when the baseline representation struggles to achieve high precision values. In addition, the aggregation strategy often improves the quality of the top-ranked positions, leading to competitive Mean Average Precision (MAP) and Precision values across different datasets and feature extractors. These findings suggest that combining projection-based and rank-based manifold learning strategies through rank aggregation can provide complementary contextual information for image retrieval tasks.