Domain-specific versus general large language models: a review and empirical benchmark in real medical texts

Abstract

The rapid adoption of Large Language Models (LLMs) has led to their widespread use as a general-purpose, fast-deployment solution across a broad range of tasks. In healthcare settings, this trend has accelerated the replacement of task-specific models with large generative architectures, often without an assessment of their efficiency or suitability for structured information extraction. While transformer-based encoder models have long been established as reliable solutions for clinical Named Entity Recognition (NER) and Information Extraction (IE), the increasing reliance on LLMs raises concerns regarding computational cost, scalability, energy consumption, and practical deployment in resource- and privacy-constrained environments. This study aimed to critically evaluate the assumption that LLMs constitute a universally superior solution for clinical NER and IE, by systematically comparing encoder-based Language Models (LMs), and LLMs in terms of extraction performance, efficiency, deployment feasibility, and environmental impact. Following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA 2020) guidelines, a scoping literature review was conducted using Scopus and Web of Science, covering peer-reviewed studies published between 2022 and 2025. In addition, a proof-of-concept case study was performed on European Portuguese clinical notes, comparing a domain-adapted encoder model (MediAlbertina-1.5B) with quantized Llama-based LLMs executed locally under identical hardware conditions. Models were evaluated using micro-averaged precision, recall, and F1-score, together with runtime, estimated energy consumption, and CO₂ emissions. The literature review showed that LLMs and transformer-based architectures were the most frequently used even thought encoder-based and domain-adapted models often perform strongly in clinical NER, particularly when recall, structured output, and computational efficiency are important. The empirical case study showed a similar pattern: MediAlbertina-1.5B achieved the highest performance (micro-F1 = 0.430), whereas all evaluated Llama variants obtained markedly lower F1-scores (≤ 0.123) due to systematic under-extraction. Despite their lower performance, the Llama models incurred 23–95 × longer runtimes and one to two orders of magnitude higher estimated energy consumption and CO₂ emissions under the tested local CPU-only setup. In this proof-of-concept benchmark on 20 European Portuguese clinical reports, the domain-adapted MediAlbertina token-classification model achieved higher recall and micro-F1 than the evaluated locally executed, quantized Llama models. The Llama models were also substantially slower and less sustainable, with higher estimated energy use and CO₂ emissions under the tested hardware and prompting configuration. These findings suggest that domain-adapted encoder models may be preferable for similar structured clinical NER settings, although larger and externally validated benchmarks are needed before generalizing to other languages, clinical corpora, model families, or deployment environments.