Scalability Analysis of Cross-modal Text-to-Image Pipelines

Abstract

Cross-modal text-to-image retrieval systems are increasingly deployed over large-scale multimedia corpora, where retrieval effectiveness must be balanced against latency, energy consumption, storage, and candidate fidelity under approximate indexing. In this paper, we present a systems-level evaluation of text-to-image retrieval pipelines under realistic deployment constraints. We study representative single-index, retrieve-and-rerank, and ensemble pipelines on MS-COCO and Flickr30K, and evaluate their scalability on a 7M-image subset of LAION using Flat, HNSW, IVFPQ, and O-IVFPQ indexes. Our results show that strong rerankers can preserve retrieval quality provided the first-stage ANN index maintains sufficient candidate fidelity, but that low-fidelity indexes create unrecoverable candidate losses. At LAION scale, O-IVFPQ provides the strongest single-index operating point, maintaining high candidate fidelity while reducing latency, energy, and storage relative to HNSW. For retrieve-and-rerank pipelines, the reranker dominates end-to-end latency and energy, making index choice less important once sufficient candidate fidelity is preserved. Overall, our study shows that scalable cross-modal retrieval requires joint reasoning about pipeline architecture, candidate fidelity, and systems cost.