Skip to content
Open access

Performance comparison of large language models in interpreting clinical guidelines for migraine prevention: A multidimensional analysis

Sep 2026 · Neurology Asia · 0 citations

TL;DR

While all four models exhibited comparable adherence to guideline-based content, DeepSeek-R1 demonstrated superior performance in supplementary informational value and readability, highlighting the importance of evaluating LLMs not only for accuracy but also for their capacity to enhance clinical communication and decision support.

Abstract

Background & Objective: Large language models (LLMs) such as DeepSeek-R1, Gemini-2.5 Pro, ChatGPT-5 Thinking, and Grok-4 Expert are increasingly applied in medical contexts, yet their reliability in evidence-based clinical domains like migraine prophylaxis remains uncertain. This study aimed to compare the performance of four leading LLMs in interpreting and applying the International Headache Society’s global practice recommendations for preventive pharmacological treatment of migraine. Methods: Sixteen standardized clinical scenario questions derived from the IHS guideline were presented identically to each model. Responses were evaluated by blinded expert raters across five dimensions—Accuracy, Overconclusiveness, Supplementary Value, Incompleteness, and Readability—using a 10-point Likert scale. Readability was further analyzed using composite indices from readabilityformulas.com. Results: No significant inter-model differences were observed in Accuracy (P = 0.856), Overconclusiveness (P = 0.400), or Incompleteness (P = 0.531). However, DeepSeek-R1 provided significantly more Supplementary Information than Gemini-2.5 Pro (P = 0.010) and Grok-4 Expert (P = 0.030). Readability analysis further revealed substantial variation across models (P < 0.001), with DeepSeek-R1 generating the most accessible outputs. Conclusion: While all four models exhibited comparable adherence to guideline-based content, DeepSeek-R1 demonstrated superior performance in supplementary informational value and readability. These findings highlight the importance of evaluating LLMs not only for accuracy but also for their capacity to enhance clinical communication and decision support.

Read PDF

Similar papers

Open access Sep 2026

A Multidimensional Evaluation of Large Language Model Responses to the 2024 ESC Hypertension Guidelines: A Comparative Study

Although overall performance was comparable, response consistency represents an additional dimension that should be considered when evaluating LLMs for guideline-based clinical applications, and future studies incorporating broader clinical scenarios and updated LLM versions are warranted.

F. Böyük, Aysun Karahan Gün, İsmail Polat Canbolat et al. · 0 citations
Open access Sep 2026

A structured exploratory multidisciplinary evaluation of four large language models responding to frequently asked patient questions about pregabalin

Significant limitations persist in safety warning compliance, response comprehensiveness, and hallucination rate for selected basic pharmaceutical questions, and significant limitations persist in safety warning compliance, response comprehensiveness, and hallucination rate.

Y. L. T. Bayala, I. A. Tinni, Thierry Boris Wend-Yam Yaméogo et al. · 0 citations
Open access Sep 2026

Public-facing health information on achalasia generated by large language models: a multidimensional comparative study

Current LLMs provided generally accurate and often useful answers to public questions about achalasia, however, some differences were found in safety, empathy, transparency, information quality, and readability.

Jun-Zheng Li, Pan Zhou, Man Yang et al. · 0 citations
Open access Sep 2026

EMHA migraine scoring system: A novel, valid, and reliable metric to capture severity, align with patients, and facilitate communication

Background Available instruments to quantify migraine-related disability, such as the Headache Impact Test (HIT-6) and Migraine Disability Assessment Test (MIDAS), have certain limitations (e.g., recall bias, overlooking interictal burden), and do not always facilitate patient-doctor communication. We developed a migra...

P. Pozo-Rosich, E. Caronna, E. R. de la Torre et al. · 0 citations

Guideline-Based Evaluation of Large Language Models in Psoriasis Treatment.

Current LLMs can match or exceed dermatologists in completeness of guideline-based systemic psoriasis recommendations, particularly in low-risk contexts, however, they remain more prone to critical safety errors in complex, high-stakes scenarios.

K. Xia, L. Min, Dan Jian · 0 citations
Open access Sep 2026

Multidimensional Evaluation of Guideline-Based Generative Artificial Intelligence Responses in Traditional Chinese: A Ménière’s Disease Study

Background: The reliability of generative artificial intelligence for Chinese medical information remains uncertain. This study evaluates the concordance and linguistic performance of three generative artificial intelligence systems in generating Chinese information on Ménière’s disease against clinical practice guidel...

Mien-Jen Lin, Yun-Chiao Wen, Li-Chun Hsieh et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.