Large Language Models for Information Extraction from Neurosurgical Clinical Notes: A Benchmark Against Human Auditors

Abstract

Retrospective clinical research depends on reliable extraction of structured variables from unstructured medical documentation, but manual chart abstraction is slow and vulnerable to inter-rater variability. We compared zero-shot extraction of clinically relevant variables from German neurosurgical records by large language models (LLMs) with extraction by two human auditors. We analyzed 881 de-identified clinical notes from 211 patients undergoing microvascular decompression and extracted 15 predefined binary variables across three domains: preoperative symptoms, postoperative complications, and disease course. Six LLMs were assessed, including cloud-based models (GPT-4o, o3-mini, Claude Sonnet 3.5, and Gemini 2.0 Flash) and locally hosted models (llama 3.3 and deepseek-r1), with three runs per patient record. Outputs were compared with two independent human auditors against a clinician-adjudicated reference standard. Cloud-based models had higher observed F1 scores than the local models across all three domains. GPT-4o achieved an F1 score of 0.95 for preoperative symptoms; GPT-4o, o3-mini, and Gemini 2.0 Flash reached F1 scores of 0.65–0.68 for postoperative complications; and Sonnet 3.5 achieved an F1 score of 0.95 for disease course. These findings suggest that zero-shot LLM prompting could be a scalable approach for clinical information extraction while emphasizing persistent limitations in rare-event detection and privacy-sensitive deployment.