Reproducible Retrieval Evaluation: A Workbench for Lexical, Semantic, and Hybrid Search

Abstract

I built this research software to compare five search configurations: SQLite FTS5 BM25, BGE-small-en-v1.5, all-MiniLM-L6-v2, and lexical fusion with each embedding model using reciprocal rank fusion. The workbench measures which relevant records are retrieved, how well they are ranked, and how long queries take. It checks complete returned records, distinguishes failed runs from valid measurements, and produces inspectable results and comparisons. The synthetic dataset contains 300 corpus records, 291 indexed records, and 188 labeled queries. Query-specific eligible pools range from 1 to 43 records, with a median of 36. The recorded runs use a development-selected fixed configuration on known development and held-out query splits. BGE gives the strongest held-out ranking quality by nDCG@10 and MRR. MiniLM has slightly higher held-out recall@10 and lower observed query latency in both held-out passes. Equal-weight lexical fusion has mixed effects and does not consistently improve the semantic methods. I interpret these findings within the small synthetic candidate pools and the measured Windows CPU setting. Pinned dependency versions and model revisions support reproduction on the tested Windows CPython 3.13 x64 platform. This software includes an accompanying technical report with the methods, measured results, failure cases, and reproducibility instructions. AI assistance: OpenAI Codex and ChatGPT—software, synthetic data and labels, analysis, figures, and drafting; Anthropic Claude—editing and review.