Large language models as digital libraries: a multi-benchmark and multi-model study
TL;DR
Querying LLMs as digital libraries is feasible, but its effectiveness depends on model strength, deployment conditions, dataset structure, and execution strategy, and Galois remains valuable when relational discipline and controlled query execution are required.
Abstract
Large Language Models (LLMs) are increasingly used as flexible access layers for information-rich collections, raising the question of whether they can also support structured access to cultural heritage data as if they were digital libraries. This work investigates that question through Galois, a database-inspired system that treats the LLM as a noisy storage layer. We evaluate six querying modes: direct natural language prompting, direct SQL prompting, and four Galois strategies that combine LLM-facing tuple retrieval with deterministic relational processing. The evaluation uses two cultural heritage benchmarks: Paintings Extended, based on paintings data, and a new benchmark based on Europeana metadata. For both benchmarks, we define fourteen query templates with ten variants each. The main experiments use GPT-4o-mini, Granite 3.3 8B, and Llama 3 70B, with an additional diagnostic analysis of two GPT-5 deployments. We complement the benchmark metrics with a stratified manual audit of 60 unique disagreements. Performance is strongly shaped by model capability, deployment behavior, dataset structure, and execution strategy. Direct natural language prompting is the strongest and most robust baseline, while direct SQL prompting becomes nearly equivalent on the strongest open models. Among the structured strategies, only GaloisA and GaloisF become meaningfully competitive, especially on the flatter Europeana benchmark. In the audited sample, 35 disagreements were model-originated failures, 21 reflected reference-coverage or metadata-normalization effects, and four remained indeterminate. The additional GPT-5 deployment analysis reveals frequent empty outputs and provider-level constraints. These diagnostic results characterize the specific tested deployments and must not be interpreted as a general ranking of GPT-5 model capability. Querying LLMs as digital libraries is feasible, but its effectiveness depends on model strength, deployment conditions, dataset structure, and execution strategy. Direct prompting is already strong for exploratory cultural heritage access, while Galois remains valuable when relational discipline and controlled query execution are required.