It is shown that current large language models generally perform poorly on most of the safety tests, regardless of their safety alignment mechanisms, which underscores the crucial need for ongoing human oversight and the implementation of strict safety guardrails before deploying these tools in clinical practice.
Abstract
The remarkable capabilities of large language models make them increasingly compelling for use in real-world healthcare applications. However, the risks associated with using these artificial intelligence systems in medicine are not systematically understood. The aim of this study is to characterize these risks by applying five key principles for safe and trustworthy medical artificial intelligence: truthfulness, resilience, fairness, robustness, and privacy.
We introduce MedGuard-Bench: a safety benchmark featuring one thousand expert-verified questions covering ten specific aspects of our five core principles. We use this comprehensive corpus to systematically evaluate sixteen commonly used large language models, assessing their safety and reliability in medical contexts.
We show that current large language models generally perform poorly on most of our safety tests, regardless of their safety alignment mechanisms. Our evaluation demonstrates that these models fall significantly short when compared to the high performance and reliability of human physicians.
Despite reports indicating that advanced large language models can match or exceed human performance in various medical tasks, this study reveals a significant safety gap in current technology. This underscores the crucial need for ongoing human oversight and the implementation of strict safety guardrails before deploying these tools in clinical practice.
The Review examines rapid LLM adoption in clinical care, outlining emerging security and safety risks across development stages, key protective layers, clinically relevant threats and current mitigation responsibilities in a single integrated framework.
J. Clusmann, O. Freyer, Max Ostermann et al.· Nature· 1 citation
It is shown that while LLMs exhibit strong recall, they often perform poorly at providing valid scientific articles and evidence for support and at rejecting unsupported hypotheses, highlighting the need for rigorous evaluation before using LLMs for search and retrieval in healthcare settings.
Safiyyah Ahmed, Abrar Ansari, M. Islam et al.· 0 citations
This article aims to provide a pragmatic basis for designing and executing rigorous evaluations of healthcare LLMs by describing underlying concepts and potential pitfalls, while emphasizing the importance of aligning evaluation methods with the research question.
S. E. McKinney, Phuc Vu, Samuel A. Justice et al.· arXiv.org· 0 citations
Abstract The flexibility of generative artificial intelligence (AI), and specifically large language models (LLMs), holds promise to transform healthcare by offering new tools to improve patient care and enhance clinician efficiency. However, this flexibility introduces new challenges, as LLM applications might inadver...
J. Workum, Sade J. Krijgsman, I. Vajda et al.· BMJ digital health & AI· 1 citation
In this single-centre proof-of-concept, aggregate accuracy was insufficient for safety characterisation: models with indistinguishable mean accuracy carried different catastrophic-failure tiers and reproducibility profiles.
Fang-Yan Liu, Zhi Liu, Xiaolu Fei et al.· Journal of medical systems· 0 citations
A comprehensive framework for transforming LLMs into trustworthy medical specialists is presented, and the evolution toward agentic AI systems that orchestrate them for autonomous, collaborative clinical decision-making is explored.
Kiduk Kim, Jeong Min Song, Dong Yeong Kim et al.· Cell Reports Medicine· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.