Benchmarking large language models for question answering on German clinical practice guidelines
Abstract Objectives Clinical practice guidelines are a cornerstone of evidence-based medicine, yet their implementation in routine care remains inconsistent. Large language models (LLMs), particularly with Retrieval-Augmented Generation (RAG), have shown strong performance in medical question answering, but their ability to use knowledge from German-language guidelines has not been systematically evaluated due to a lack of a dedicated benchmark. We therefore developed such a benchmark and evaluated guideline-based question answering with different LLMs and retriever configurations. Methods We developed cpgQA-DE, an expert-validated benchmark dataset of 200 multiple-choice questions derived from 10 current German clinical practice guidelines across five specialties. All questions were reviewed for correctness, relevance and complexity. The dataset includes case-based and knowledge-based questions with metadata on guideline source, specialty, relevance and difficulty. We evaluated LLM performance using a RAG-based pipeline built on a corpus of German guidelines, comparing multiple model–retriever combinations. Results RAG integration substantially improved accuracy across all tested models and for most retrievers. The best-performing configuration, GPT-5 combined with the multilingual-e5-large retriever, achieved an accuracy of 95%. Notably, the open-weight model gpt-oss-120b reached 90% accuracy when used with RAG. Conclusions cpgQA-DE enables systematic and reproducible offline evaluation of guideline-aware question answering systems in the German healthcare context. The observed performance gains with RAG support the use of LLMs augmented with quality-assured external knowledge. Such systems may help bridge the evidence-practice gap by providing guideline-based recommendations at the point of care, while strong performance of open-weight models suggests potential for on-premises deployment in privacy-sensitive clinical environments.