Newer models may be more robust, but multiple-choice accuracy remains an incomplete measure of clinical reasoning robustness, as all evaluated LLMs showed a significant robustness gap after NOTA replacement.
Abstract
Background: Whether benchmark performance reflects robust clinical reasoning rather than surface-level pattern recognition remains uncertain. We evaluated the robustness of state-of-the-art large language models (LLMs) on nephrology board renewal questions using "None of the other answers" (NOTA) substitution. Methods: From 210 Japanese Society of Nephrology board renewal questions (2014-2023), two nephrologists independently reviewed all items. Questions in which NOTA became the sole correct answer after replacement were included, yielding 145 validated questions. GPT-5, GPT-4o, Gemini 2.5 Pro, and Gemini 2.0 Flash were evaluated via application programming interfaces under default settings. The primary endpoint was accuracy, and paired differences were assessed using the exact two-sided McNemar test. Results: Accuracy was significantly lower after NOTA substitution for all models: GPT-4o, 66.21% to 19.31% (drop, 46.90 percentage points [pp]); GPT-5, 87.59% to 73.10% (14.48 pp); Gemini 2.0 Flash, 58.62% to 31.03% (27.59 pp); and Gemini 2.5 Pro, 86.90% to 55.86% (31.03 pp); all P < .001. GPT-5 showed the smallest decline and the highest accuracy in both versions. Conclusions: All evaluated LLMs showed a significant robustness gap after NOTA replacement. Newer models may be more robust, but multiple-choice accuracy remains an incomplete measure of clinical reasoning robustness.
Overall, advanced prompting markedly improved model performance, and top-tier LLMs demonstrated robust interpretive capability, while caution is needed for variables with complex clinical semantics such as blood pressure.
Jiwon You, Hangsik Shin· npj Digital Medicine· 0 citations
The release of the 2025 American Society of Regional Anesthesia (ASRA) 5th edition guidelines for regional and neuraxial anesthesia procedures for patients receiving antithrombotic medications introduced complex, patient-specific hold times and resumption protocols. As clinicians increasingly utilize large language models (LLMs) as clinical decision support tools, the reliability of these models remains largely unvalidated. This study evaluates the accuracy of two of the foremost LLMs, ChatGPT (OpenAI, San Francisco, CA) and Google Gemini (Google DeepMind, London, UK), in adhering to these new gold-standard safety guidelines. Twenty-five standardized clinical vignettes were developed. Each vignette featured a patient on a specific anticoagulant (e.g., rivaroxaban, apixaban, dabigatran) requiring a neuraxial or regional anesthetic procedure (stratified by high-risk vs. low-risk). Variables included renal function, dose frequency, and procedural urgency. Prompts were submitted to the latest publicly available ChatGPT and Google Gemini models with separate instructions to provide hold and resumption times. LLMs were queried to ensure familiarity with 2025 ASRA guidelines prior to submission of prompts. Responses were graded against the 2025 ASRA guidelines by independent reviewers. Response adherence was categorized as: 1. concordant (100% match); 2. conservative error (LLM recommended time was longer than required); 3. dangerous error (recommended time was shorter than required, a critical safety violation); or 4. omission (no specific timeframe provided). ChatGPT achieved a concordance rate of 64%, compared to Gemini at 62%. However, the models displayed distinct error profiles. ChatGPT produced "dangerous errors" in 20% of evaluations and failed to specify a time in 16% of cases. In contrast, Gemini’s dangerous error rate was lower at 12%, but it demonstrated a significant "conservative error" rate of 22%, compared to 0% for ChatGPT. A chi-square test indicated that the difference in dangerous error rates between the two models was not statistically significant. Regardless, notable differences in error character and response completeness were observed. Gemini was more consistent in providing specific timeframes in 96% of prompts compared to 84% for ChatGPT. Variability of responses between users was determined via a chi-square test of homogeneity and was found to be statistically significant only for Gemini. While both models demonstrated moderate guideline awareness, their failure modes differed meaningfully. ChatGPT's errors were predominantly dangerous underestimations and omissions, while Gemini exhibited a conservative bias, overestimating hold times when a match was not achieved. Significant limitations exist regarding generalizability of these results. Only two major LLM models were tested. Other, more clinically oriented models exist, and newer versions of both ChatGPT and Gemini are consistently being released. These may improve LLM adherence to clinical guidelines. Despite these limitations, this study highlights the dangers of utilizing LLMs for periprocedural anticoagulation decision-making in regional and neuraxial anesthesia. Both models produced potentially dangerous recommendations in 12-20% of scenarios. Ongoing evaluation of LLM adherence to evolving guidelines remains essential, and clinicians must exercise extreme caution when utilizing these tools in patient care.
Conner M. Willson, Birpartap S. Thind, Jay Srinivas et al.· Cureus· 0 citations
While unprompted models exhibit marked baseline linguistic and quality variations, the strategic integration of robust prompt engineering successfully enforces the temporal stability and clarity required for reliable digital public health communication.
Ömer Önal, Suzan Temiz Bekce· Scientific Reports· 0 citations
Timely recognition of sepsis remains difficult when early physiological abnormalities are subtle or incomplete. This
study examined whether general-purpose large language models could discriminate sepsis risk from an initial ICU vital-sign
snapshot as effectively as established clinical scoring approaches. We performed a retrospective benchmark using the deidentified PhysioNet Sepsis Prediction Dataset. The eligible cohort included 39,234 adults, of whom 2,733 were sepsis-positive.
Two zero-shot language models and two modified clinical scores were evaluated on the same patients. Discrimination was
measured by the area under the receiver operating characteristic curve (AUROC), with bootstrap confidence intervals and
DeLong tests for paired comparisons. AUROC was 0.597 for GPT-5.5, 0.592 for modified NEWS2, 0.591 for Claude Sonnet 5,
and 0.570 for modified SOFA. Neither language model differed significantly from modified NEWS2, whereas both produced
higher AUROC values than modified SOFA. A pre-onset-only sensitivity analysis, restricted to patients whose snapshot preceded
their first positive sepsis label (achieved median lead time 39 hours), showed all four estimators converging (AUROC 0.579–
0.591) with no significant pairwise differences, indicating the primary-analysis gap over SOFA was partly attributable to postonset records. A stricter analysis limited to a minimum 6-hour lead time reversed the ranking: the SOFA-derived score obtained
the highest AUROC (0.597), with the LLMs and NEWS2-derived score converging lower (0.573–0.578), again with no
significant pairwise differences. Their alerting behaviour was not interchangeable: GPT-5.5 produced a more balanced
sensitivity-specificity profile, while Claude Sonnet 5 identified more positive cases at the cost of additional false alerts. These
findings show that zero-shot language models can provide discrimination comparable to the strongest modified clinical score in
this dataset. The modest absolute AUROC values, use of modified scores, and retrospective single-dataset design mean that the
models should be considered complementary research tools, not stand-alone clinical detectors.
Anvit More, Vishala Bodetti, Kishan Gor et al.· International Journal for Re...· 0 citations
Large Language Models (LLMs) are increasingly used by clinicians and patients for medical queries, yet their accuracy and safety at the specialist level in hematology remain insufficiently characterised. We benchmarked ten frontier proprietary and open-weight LLMs across two generations on 1,477 board-style hematology multiple-choice questions (MCQs) derived from five educational datasets spanning nine disease areas and six clinical skill domains, including text-only and multimodal case vignettes. Claude Opus 5 had the highest mean accuracy (92.7% text, 76.9% multimodal), followed closely by Gemini-3.1 Pro (91.4% and 78.7%), Gemini-3.6 Flash (91.0% and 74.8%) and GPT-5.6 Sol (89.9% and 76.7%). Accuracy significantly correlated with model size both for text-only and multimodal MCQs. Between model generations, the largest improvements in accuracy were seen for open-weight models whereas proprietary models showed only marginal gains. In error analysis, top-performing models exhibited highly concordant failure patterns, suggesting shared limitations on challenging cases. Frontier LLMs exhibit substantial specialist hematology knowledge across diverse subspecialist domains and clinical skill sets. Yet, despite high accuracy on board-style questions in hematology, continuous expert-on-the-loop output monitoring is paramount.
M. Radoynova, M. Benouis, F. Schulze et al.· medRxiv· 0 citations
Although contemporary LLMs increasingly reflect medical consensus for CNS metastases, inconsistent reliability remains a concern, underscoring the need for caution in patient use.
Michael Fiorino, Mei Hainline, Tanay Poddar et al.· Neuro-Oncology Advances· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
