Benchmarking large language models for clinical data extraction from Portuguese medical notes in a university hospital

Abstract

Extracting structured data from electronic health records (EHRs) remains a major challenge, particularly in non-English and resource-constrained healthcare systems. This study benchmarks multiple large language models (LLMs) for the automated extraction of structured clinical variables from Portuguese-language medical notes under limited computational resources. We evaluated five LLMs (GPT-4o mini, DeepSeek-V3, Mixtral-8x7B, LLaMA 8B, and Qwen-32B) against a manually curated dataset of cardiology and infectiology outpatient records. Models were deployed in quantized versions to optimize computational efficiency. Outputs were compared with human annotations using F1 score, balanced accuracy, and recall. Among the tested models, Qwen-32B achieved the highest performance in both the infectiology domain (balanced accuracy = 0.91 [0.07]) and cardiology domain (balanced accuracy = 0.89 [0.07]). Performance varied by clinical variable, with better results for frequently and consistently documented conditions (e.g., diabetes) and lower accuracy for complex or infrequent variables (e.g., tumors). Extraction time ranged from 0.9 to 24.2 minutes per patient, depending on clinical domain and model. These findings support the feasibility of applying LLM-based natural language processing tools in resource-limited, non-English healthcare settings. Future research should assess emerging high-parameter models and explore additional clinical domains.