Adaptive LLM Selection for Question Answering over Historical Digital Libraries

Abstract

Historical digital libraries hold millions of scanned pages spanning centuries of human knowledge. However, high-fidelity content accessibility through automated question answering remains largely unsolved. Large language models offer a promising path but historical documents pose particular challenges such as OCR noise, archaic spelling, mixed layouts and domain shift from modern training corpora. Moreover, large language models exhibit sharply divergent competence across these challenges. Selecting the wrong model silently degrades retrieval quality at scale, with no easy way for digital library practitioners to know which model to trust for a given document. To address this challenge, we investigate adaptive LLM selection as a core problem for historical digital libraries. We construct a large-scale benchmark of 84,441 multiple-choice questions derived from 5,941 scanned pages of bilingual English and Chinese historical periodicals. We systematically evaluate 50 open-source LLMs ranging from 0.5B to 9B parameters, producing a page-level binary correctness matrix of 50 × 5, 941 entries. Our analysis reveals that model performance is highly page-dependent: the per-page oracle accuracy reaches 86.1% while the static best model achieves only 42.8%. We further show that our routing models include the oracle-best model in the top-3 candidate models for more than 61% of pages, offering a practical path toward efficient deployment. Gains are largest on the most challenging pages, including tabular content, Chinese-language text and hard pages where the dominant model consistently fails. Expert validation further confirms that adaptive selection decisions derived from automatic MCQs are consistent with historian judgement. Our benchmark and methodology establish a reproducible framework for studying and deploying adaptive LLM selection in historical digital library pipelines.