Sep 2026· Journal of Cardiovascular Development and Disease· Vol 13· 0 citations· 21 references
Medicine
TL;DR
Although overall performance was comparable, response consistency represents an additional dimension that should be considered when evaluating LLMs for guideline-based clinical applications, and future studies incorporating broader clinical scenarios and updated LLM versions are warranted.
Abstract
Large language models (LLMs), including ChatGPT, Gemini, and DeepSeek, are increasingly used in medicine; however, their performance across multiple clinically relevant domains remains incompletely understood. This cross-sectional comparative study evaluated 225 responses generated by three LLMs to 75 clinical questions selected from the 2024 European Society of Cardiology (ESC) Hypertension Guidelines. Responses were independently assessed by two board-certified cardiologists across five predefined domains: accuracy, clinical relevance, completeness, absence of bias and misinformation, and consistency. No statistically significant differences were observed among the three models in accuracy, clinical relevance, completeness, or absence of bias and misinformation (all p > 0.05). A significant difference was identified in response consistency (p = 0.032), with post hoc analysis demonstrating a significant difference between ChatGPT and Gemini. Overall appropriateness scores did not differ significantly among the three LLMs (p = 0.227). These findings suggest that, although overall performance was comparable, response consistency represents an additional dimension that should be considered when evaluating LLMs for guideline-based clinical applications. Future studies incorporating broader clinical scenarios and updated LLM versions are warranted to further define their role in clinical decision support.
While all four models exhibited comparable adherence to guideline-based content, DeepSeek-R1 demonstrated superior performance in supplementary informational value and readability, highlighting the importance of evaluating LLMs not only for accuracy but also for their capacity to enhance clinical communication and deci...
Li Xu, Xu Qiu, Jia-Yi Deng et al.· Neurology Asia· 0 citations
Significant limitations persist in safety warning compliance, response comprehensiveness, and hallucination rate for selected basic pharmaceutical questions, and significant limitations persist in safety warning compliance, response comprehensiveness, and hallucination rate.
Y. L. T. Bayala, I. A. Tinni, Thierry Boris Wend-Yam Yaméogo et al.· Discover Artificial Intellig...· 0 citations
Although the four LLMs generally provide accurate and pertinent information regarding type 2 diabetes, enduring limits in actionability and inconsistencies among models in content completeness and understandability restrict their effective use in diabetic patient education.
Zhaoxia Huang, Yuxin Bai, Jun-Yue Luo et al.· Frontiers in Public Health· 0 citations
BACKGROUND
large language models (LLMs) are increasingly used for public health information, but their performance in nutrition advice for cirrhosis remains uncertain. We investigated four LLMs in answering public questions about cirrhosis nutrition across safety, accuracy, empathy, information reliability and quality,...
Jun-Zheng Li, Ying-Jie Wu, Man Yang et al.· Nutrición Hospitalaria· 0 citations
LLM-based chatbots can provide acceptable surgical guidance for straightforward primary surgical cases, but their utility is limited in high-risk or complex clinical settings, and their rationale quality was significantly lower in complex scenarios than in primary ones.
Müge Toprak, Büşra Yılmaz Tuğan, N. Yüksel· Seminars in Ophthalmology· 0 citations
Clinical guidelines may sometimes contain divergent treatment recommendations, creating uncertainty in therapeutic decision-making. To evaluate the performance of four large language models (LLMs) (GPT-4o, Claude 3.7 Sonnet, Gemini 1.5 Pro, Deepseek-V3) in analyzing discrepancies between the American Society of Clini...