A scalable and semantic pipeline for efficient video indexing and person retrieval

Abstract

Abstract The explosive growth of large-scale video archives from surveillance networks, online platforms, and personal devices has made efficient and semantically rich video retrieval a critical challenge. Existing approaches based on deep multimodal embeddings have significantly improved retrieval accuracy. However, they often lack scalability, modularity, and system-level integration with indexing and metadata management. In this work, we present a modular and scalable pipeline for semantic video indexing and retrieval, tailored to person-centric search. The proposed architecture decouples a web-based front-end from a back-end organized into two pipelines. The indexing pipeline performs video chunking, person detection and tracking, crop selection, metadata enrichment, and semantic vectorization. The retrieval pipeline supports visual, textual, and hybrid queries, including face-based matching. The system leverages YOLO11 and BoT-SORT for real-time person detection and tracking, SigLIP2 for multilingual vision-language embeddings, and InsightFace for face recognition, storing all representations in a vector database with rich, traceable metadata. We further fine-tune the SigLIP2-SO400M-Patch14-384 checkpoint on a curated mixture of person-centric image-text datasets and evaluate the resulting model on the RSTPReid benchmark. Experimental results show that our approach achieves state-of-the-art performance on the RSTPReid benchmark under the considered setting. In particular, it achieves competitive Recall@k performance with respect to recent text-based person search methods and significantly improves mean Average Precision, reaching 0.68 against a best competing value of 0.54.