A Hybrid Search Engine Combining Classical Information Retrieval with Pluggable Semantic Search
TL;DR
A hybrid search engine whose coreretrieval, ranking, and fusion components, an inverted index,TF-IDF, Okapi BM25, dense vector search, and score fusion, are implemented directly in Python rather than delegated to anexisting search or vector-database framework, so that the tradeoff can be measured directly.
Abstract
Most search features in production today sit on topof a mature platform, typically Elasticsearch, Apache Lucene,or a managed vector database, and the algorithms doing theactual retrieval work are rarely touched by the engineer usingthem. This paper describes a hybrid search engine whose coreretrieval, ranking, and fusion components, an inverted index,TF-IDF, Okapi BM25, dense vector search, and score fusion,are implemented directly in Python rather than delegated to anexisting search or vector-database framework, so that the tradeoffsthose platforms hide can be measured directly. Standardsupporting libraries (a web framework, document parsers, anoptional embedding backend) are used where they are incidentalto that goal, but none of them performs the retrieval or rankingwork itself. The system pairs classical lexical retrieval, an invertedindex supporting Boolean and phrase queries, TF-IDF, and OkapiBM25, with a pluggable semantic layer built on dense embeddingsand cosine-similarity vector search. The two signals are combinedthrough a normalized linear fusion into a single hybrid rankingfunction, and the retrieval stack is further extended into a smallretrieval-augmented generation (RAG) pipeline. The ranking,fusion, and generation components are implemented and testedat the library and benchmark level; the deployed command-lineand HTTP interfaces currently serve BM25 only, a boundary thispaper states explicitly rather than blurs. Development proceededacross eleven incremental phases, moving from custom datastructures (hash tables, tries, heaps, graphs) through the rankingalgorithms, a custom binary persistent index format, parallel andasynchronous performance work, a breadth-first web crawler, aFastAPI backend, and a React frontend. Of 414 automated unittests, 408 pass (98.6%); the six that fail trace back to a genuine,reproducible infinite loop in the Knuth–Morris–Pratt substringmatchingroutine, documented here as an engineering findingrather than fixed quietly and forgotten. A retrieval-effectivenessbenchmark, run over a small 15-document relevance-judged corpuswith only four queries (too small a scale to support a strongeffectiveness claim), shows BM25 (NDCG@5 = 0.964) ahead ofTF-IDF (0.910) and every tested hash-based hybrid configuration.A follow-up sweep over α ∈ {0.1, 0.3, 0.5, 0.7, 0.9} withthe dependency-free hash embedding provider shows NDCG@5rising monotonically with α, from 0.753 at α = 0.1 to abest of 0.917 at α = 0.9, still below BM25’s 0.964; a hashembedding-only run (NDCG@5 = 0.753) and a Reciprocal RankFusion baseline combining BM25 with hash-based semanticscores (NDCG@5 = 0.866) likewise fall short of BM25 alone.On this corpus, then, no tested hybrid or fusion configurationimproves on plain BM25. A separate attempt to evaluate atrained sentence-embedding model (all-MiniLM-L6-v2) wasblocked by the execution environment, insufficient disk spacefor the CUDA/PyTorch dependency stack, missing CUDA sharedlibraries in a reduced-dependency install, and blocked networkaccess to the model hub, so this paper draws no conclusion aboutreal dense semantic retrieval and reports the blocker itself as areproducible engineering finding rather than a result. A separateparallel-indexing benchmark confirms, on hardware with a singleavailable CPU core, that the multiprocessing speedups reportedin the project’s internal documentation (3.14× on a multicoremachine) turn into a slowdown (0.77×) once no physicalparallelism is available, an empirical illustration of Amdahl’slaw. Taken together, the results position the system as a referenceimplementation for understanding hybrid retrieval and systemsengineering, validated to the explicit scope described in Table X,and not as a production-ready replacement for existing searchinfrastructure.