While all four models exhibited comparable adherence to guideline-based content, DeepSeek-R1 demonstrated superior performance in supplementary informational value and readability, highlighting the importance of evaluating LLMs not only for accuracy but also for their capacity to enhance clinical communication and decision support.
Abstract
Background & Objective: Large language models (LLMs) such as DeepSeek-R1, Gemini-2.5 Pro, ChatGPT-5 Thinking, and Grok-4 Expert are increasingly applied in medical contexts, yet their reliability in evidence-based clinical domains like migraine prophylaxis remains uncertain. This study aimed to compare the performance of four leading LLMs in interpreting and applying the International Headache Society’s global practice recommendations for preventive pharmacological treatment of migraine.
Methods: Sixteen standardized clinical scenario questions derived from the IHS guideline were presented identically to each model. Responses were evaluated by blinded expert raters across five dimensions—Accuracy, Overconclusiveness, Supplementary Value, Incompleteness, and Readability—using a 10-point Likert scale. Readability was further analyzed using composite indices from readabilityformulas.com.
Results: No significant inter-model differences were observed in Accuracy (P = 0.856), Overconclusiveness (P = 0.400), or Incompleteness (P = 0.531). However, DeepSeek-R1 provided significantly more Supplementary Information than Gemini-2.5 Pro (P = 0.010) and Grok-4 Expert (P = 0.030). Readability analysis further revealed substantial variation across models (P < 0.001), with DeepSeek-R1 generating the most accessible outputs.
Conclusion: While all four models exhibited comparable adherence to guideline-based content, DeepSeek-R1 demonstrated superior performance in supplementary informational value and readability. These findings highlight the importance of evaluating LLMs not only for accuracy but also for their capacity to enhance clinical communication and decision support.
Although overall performance was comparable, response consistency represents an additional dimension that should be considered when evaluating LLMs for guideline-based clinical applications, and future studies incorporating broader clinical scenarios and updated LLM versions are warranted.
F. Böyük, Aysun Karahan Gün, İsmail Polat Canbolat et al.· Journal of Cardiovascular De...· 0 citations
Significant limitations persist in safety warning compliance, response comprehensiveness, and hallucination rate for selected basic pharmaceutical questions, and significant limitations persist in safety warning compliance, response comprehensiveness, and hallucination rate.
Y. L. T. Bayala, I. A. Tinni, Thierry Boris Wend-Yam Yaméogo et al.· Discover Artificial Intellig...· 0 citations
Current LLMs provided generally accurate and often useful answers to public questions about achalasia, however, some differences were found in safety, empathy, transparency, information quality, and readability.
Jun-Zheng Li, Pan Zhou, Man Yang et al.· Frontiers in Public Health· 0 citations
Background Available instruments to quantify migraine-related disability, such as the Headache Impact Test (HIT-6) and Migraine Disability Assessment Test (MIDAS), have certain limitations (e.g., recall bias, overlooking interictal burden), and do not always facilitate patient-doctor communication. We developed a migra...
P. Pozo-Rosich, E. Caronna, E. R. de la Torre et al.· Cephalalgia· 0 citations
Current LLMs can match or exceed dermatologists in completeness of guideline-based systemic psoriasis recommendations, particularly in low-risk contexts, however, they remain more prone to critical safety errors in complex, high-stakes scenarios.
K. Xia, L. Min, Dan Jian· Clincal and Experimental Der...· 0 citations
Background: The reliability of generative artificial intelligence for Chinese medical information remains uncertain. This study evaluates the concordance and linguistic performance of three generative artificial intelligence systems in generating Chinese information on Ménière’s disease against clinical practice guidel...