Skip to content
Open access

Assessing safety and trustworthiness of large language models in medicine

Sep 2026 · Communications Medicine · 0 citations

TL;DR

It is shown that current large language models generally perform poorly on most of the safety tests, regardless of their safety alignment mechanisms, which underscores the crucial need for ongoing human oversight and the implementation of strict safety guardrails before deploying these tools in clinical practice.

Abstract

The remarkable capabilities of large language models make them increasingly compelling for use in real-world healthcare applications. However, the risks associated with using these artificial intelligence systems in medicine are not systematically understood. The aim of this study is to characterize these risks by applying five key principles for safe and trustworthy medical artificial intelligence: truthfulness, resilience, fairness, robustness, and privacy. We introduce MedGuard-Bench: a safety benchmark featuring one thousand expert-verified questions covering ten specific aspects of our five core principles. We use this comprehensive corpus to systematically evaluate sixteen commonly used large language models, assessing their safety and reliability in medical contexts. We show that current large language models generally perform poorly on most of our safety tests, regardless of their safety alignment mechanisms. Our evaluation demonstrates that these models fall significantly short when compared to the high performance and reliability of human physicians. Despite reports indicating that advanced large language models can match or exceed human performance in various medical tasks, this study reveals a significant safety gap in current technology. This underscores the crucial need for ongoing human oversight and the implementation of strict safety guardrails before deploying these tools in clinical practice.

Read PDF

Similar papers

Review Aug 2026

Safety and security of large language models in healthcare

The Review examines rapid LLM adoption in clinical care, outlining emerging security and safety risks across development stages, key protective layers, clinically relevant threats and current mitigation responsibilities in a single integrated framework.

J. Clusmann, O. Freyer, Max Ostermann et al. · 1 citation
#artificial intelligence Review Aug 2026

Medical Causal Hypothesis Verification with Large Language Models

It is shown that while LLMs exhibit strong recall, they often perform poorly at providing valid scientific articles and evidence for support and at rejecting unsupported hypotheses, highlighting the need for rigorous evaluation before using LLMs for search and retrieval in healthcare settings.

Safiyyah Ahmed, Abrar Ansari, M. Islam et al. · 0 citations
#artificial intelligence Review Sep 2026

A primer on evaluation methods for large language models in healthcare

This article aims to provide a pragmatic basis for designing and executing rigorous evaluations of healthcare LLMs by describing underlying concepts and potential pitfalls, while emphasizing the importance of aligning evaluation methods with the research question.

S. E. McKinney, Phuc Vu, Samuel A. Justice et al. · 0 citations
Review Open access Oct 2026

When are large language models medical devices? Navigating the MDR and AI Act for healthcare applications

Abstract The flexibility of generative artificial intelligence (AI), and specifically large language models (LLMs), holds promise to transform healthcare by offering new tools to improve patient care and enhance clinician efficiency. However, this flexibility introduces new challenges, as LLM applications might inadver...

J. Workum, Sade J. Krijgsman, I. Vajda et al. · 1 citation
Open access Sep 2026

Development of a Framework for Evaluating Large Language Model Safety and Reliability: a Proof-of-Concept Evaluation

In this single-centre proof-of-concept, aggregate accuracy was insufficient for safety characterisation: models with indistinguishable mean accuracy carried different catastrophic-failure tiers and reproducibility profiles.

Fang-Yan Liu, Zhi Liu, Xiaolu Fei et al. · 0 citations
Review Open access Sep 2026

Transforming large language models into medical specialists via knowledge injection.

A comprehensive framework for transforming LLMs into trustworthy medical specialists is presented, and the evolution toward agentic AI systems that orchestrate them for autonomous, collaborative clinical decision-making is explored.

Kiduk Kim, Jeong Min Song, Dong Yeong Kim et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.