BPE-Level Visual-Textual Alignment for Multi-Scene Text Retrieval

Abstract

Scene Text Retrieval (STR) aims to search images containing a given textual query within large-scale image collections. However, existing approaches are fundamentally constrained in two ways: 1) they are evaluated on narrow benchmarks that focus primarily on natural scenes; and 2) they rely on either error-prone multi-stage recognition-then-matching pipelines or localization-assisted matching strategies. To address these limitations, we introduce MuST, the first comprehensive benchmark for multi-scene and bilingual STR tasks, covering a broad range of real-world scenarios with carefully curated Chinese and English textual queries. On top of this benchmark, we propose BPE-Ret, a novel Byte-Pair Encoding (BPE)-level retrieval framework built on a simple yet powerful principle: a word is considered present in an image if and only if all of its constituent subwords are present. Concretely, BPE-Ret decomposes textual queries into BPE subwords and directly aligns them with dense visual features within a unified embedding space, thereby eliminating the need for explicit text spotting and coarse-grained word-level matching. We further enhance fine-grained alignment through two key innovations: a weighted preference learning scheme that prioritizes challenging cases to sharpen discrimination on confusable word-image pairs, and a subword inclusive-OR matching strategy that enforces constituent subword verification to enable robust retrieval beyond word-level granularity. Extensive experiments show that BPE-Ret establishes new state-of-the-art performance on both existing public STR benchmarks and our newly proposed MuST dataset, demonstrating the effectiveness and robustness of subword-level retrieval for real-world, multilingual scene text understanding.