Skip to content
Open access

A Multidimensional Evaluation of Large Language Model Responses to the 2024 ESC Hypertension Guidelines: A Comparative Study

Sep 2026 · Journal of Cardiovascular Development and Disease · Vol 13 · 0 citations · 21 references
Medicine

TL;DR

Although overall performance was comparable, response consistency represents an additional dimension that should be considered when evaluating LLMs for guideline-based clinical applications, and future studies incorporating broader clinical scenarios and updated LLM versions are warranted.

Abstract

Large language models (LLMs), including ChatGPT, Gemini, and DeepSeek, are increasingly used in medicine; however, their performance across multiple clinically relevant domains remains incompletely understood. This cross-sectional comparative study evaluated 225 responses generated by three LLMs to 75 clinical questions selected from the 2024 European Society of Cardiology (ESC) Hypertension Guidelines. Responses were independently assessed by two board-certified cardiologists across five predefined domains: accuracy, clinical relevance, completeness, absence of bias and misinformation, and consistency. No statistically significant differences were observed among the three models in accuracy, clinical relevance, completeness, or absence of bias and misinformation (all p > 0.05). A significant difference was identified in response consistency (p = 0.032), with post hoc analysis demonstrating a significant difference between ChatGPT and Gemini. Overall appropriateness scores did not differ significantly among the three LLMs (p = 0.227). These findings suggest that, although overall performance was comparable, response consistency represents an additional dimension that should be considered when evaluating LLMs for guideline-based clinical applications. Future studies incorporating broader clinical scenarios and updated LLM versions are warranted to further define their role in clinical decision support.

Read PDF

Similar papers

Open access Sep 2026

Performance comparison of large language models in interpreting clinical guidelines for migraine prevention: A multidimensional analysis

While all four models exhibited comparable adherence to guideline-based content, DeepSeek-R1 demonstrated superior performance in supplementary informational value and readability, highlighting the importance of evaluating LLMs not only for accuracy but also for their capacity to enhance clinical communication and deci...

Li Xu, Xu Qiu, Jia-Yi Deng et al. · 0 citations
Open access Sep 2026

A structured exploratory multidisciplinary evaluation of four large language models responding to frequently asked patient questions about pregabalin

Significant limitations persist in safety warning compliance, response comprehensiveness, and hallucination rate for selected basic pharmaceutical questions, and significant limitations persist in safety warning compliance, response comprehensiveness, and hallucination rate.

Y. L. T. Bayala, I. A. Tinni, Thierry Boris Wend-Yam Yaméogo et al. · 0 citations
Open access Aug 2026

Evaluating large language models using the Type 2 Diabetes Health Education guideline: a comparative analysis of ChatGPT-4.1, Claude-4.0, DeepSeek-V3, and ERNIE Bot 4.5 Turbo

Although the four LLMs generally provide accurate and pertinent information regarding type 2 diabetes, enduring limits in actionability and inconsistencies among models in content completeness and understandability restrict their effective use in diabetic patient education.

Zhaoxia Huang, Yuxin Bai, Jun-Yue Luo et al. · 0 citations
Open access Sep 2026

Performance of large language models in answering public questions about nutrition in cirrhosis: a comparative study.

BACKGROUND large language models (LLMs) are increasingly used for public health information, but their performance in nutrition advice for cirrhosis remains uncertain. We investigated four LLMs in answering public questions about cirrhosis nutrition across safety, accuracy, empathy, information reliability and quality,...

Jun-Zheng Li, Ying-Jie Wu, Man Yang et al. · 0 citations
Aug 2026

Evaluation of Large Language Model-Generated Recommendations in Glaucoma Surgical Decision-Making.

LLM-based chatbots can provide acceptable surgical guidance for straightforward primary surgical cases, but their utility is limited in high-risk or complex clinical settings, and their rationale quality was significantly lower in complex scenarios than in primary ones.

Müge Toprak, Büşra Yılmaz Tuğan, N. Yüksel · 0 citations
Open access Oct 2026

Large language models reveal a systematic preference in clinical immunotherapy guidelines: a multi-model comparative analysis

Clinical guidelines may sometimes contain divergent treatment recommendations, creating uncertainty in therapeutic decision-making. To evaluate the performance of four large language models (LLMs) (GPT-4o, Claude 3.7 Sonnet, Gemini 1.5 Pro, Deepseek-V3) in analyzing discrepancies between the American Society of Clini...

Zhen-Yu Chen, An-Qi Lin, Jun-Yi Shen et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.