Evaluation of Open-Source LLMs for Automated Subject Metadata Generation in Scholarly Workflows

Abstract

Academic libraries, repositories, and cultural heritage aggregators face growing metadata backlogs as digital collections expand. Controlled metadata assignment remains essential for discovery, browsing, and interoperability, but it is labor intensive and difficult to scale. This paper evaluates four open-weight large language models on two DPLA-derived metadata tasks: LCSH subject assignment and normalized location assignment. Using record text and selected metadata fields, we compare zero-shot prompting, retrieval-only baselines, and retrieval-augmented generation within a shared pipeline. Across both tasks, retriever configuration has a larger effect on end-to-end performance than small prompt variations, and hybrid retrieval with metadata pre-filtering disabled produces the strongest results. Exact LCSH assignment remains difficult, whereas normalized location assignment is substantially more tractable. The results support open-weight LLMs as human-reviewed metadata aids rather than as fully automatic cataloging systems, while also showing that deployment feasibility depends on institutional compute, privacy constraints, and review workflow design.