Skip to content
Review Open access

Feasibility and expert evaluation of a medical large language model in an interdisciplinary clinical board

Oct 2026 · Scientific Reports · Vol 16 · 0 citations · 13 references

Abstract

Interdisciplinary clinical boards (ICB) on chronic inflammatory diseases discuss complicated and multidisciplinary cases with inflammatory conditions among experts in their discipline. Medically specialized large language models (LLMs) are finding increasingly widespread use among physicians to answer clinical questions. LLMs offer the potential to close the gap between rapidly expanding medical knowledge and limited time of physicians by providing evidence-based summaries to clinical questions and enable “real-time” literature search. However, it is unclear if LLMs can be implemented in clinical routine of ICB. In this prospective study, 100 consecutive real-world cases were discussed in our ICB. Each case was first discussed independently of the LLM and a preliminary recommendation was documented. The clinical question was then entered in the LLM “OpenEvidence”, the answer was reviewed by the board with the opportunity to add aspects not previously raised, and the recommendation was subsequently confirmed as final. Each participating expert evaluated the answer of the LLM using a questionnaire comprising eight Likert scale (1–5) items and one free-text item. Two physicians, blinded to each other, additionally evaluated outputs for their agreement with the board’s decision. The same cases were evaluated after prompt optimization at two different time points. The 100 discussed cases resulted in 759 rater-case evaluations. A mean of 2.3 specialties were directly involved in each case. The highest score was achieved by the item “OpenEvidence provided clinically relevant information for this case.” (4.06). Recommendation for future use achieved a mean score of 3.95. Complete agreement of the LLM answer with the final recommendation of the ICB was rated in 40.2% and 50.5% of all evaluable cases (n = 97) by the two raters. When restricted to consensus cases, i.e. cases rated identically by both raters (n = 62 of 97), complete agreement was 51.6%. Outputs after prompt optimization tended to be associated with higher agreement, but did not reach statistical significance. Outputs were not stable over time. The integration of OpenEvidence into an ICB was feasible and was perceived as useful by the participating specialists, with partial or complete concordance with the final board recommendation in the majority of evaluable cases. As the study did not assess patient outcomes, clinical harm or hallucinations, no conclusions on patient safety can be drawn. The LLM was naturally limited by the information provided in the prompt. Inadequate questions or missing information led to suboptimal answers, highlighting the importance of identifying and including relevant information beforehand. Expert oversight remains essential. Model drift of LLMs may outweigh prompt optimization.

Read PDF

Similar papers

Open access Sep 2026

Large Language Models in Multidisciplinary Decision-Making for Hepatopancreatobiliary Oncology: Retrospective Comparative Feasibility Study

LLM-generated treatment recommendations demonstrated moderate alignment with MDT decisions in HPB oncology, indicating that concordance alone is insufficient for evaluating LLMs as clinical decision support tools.

Jun-Jo Sung, Eui Hyuk Chong, Incheon Kang et al. · 0 citations
Open access Oct 2026

Concordance Between Large Language Models and Multidisciplinary Tumor Board Recommendations in Treatment Allocation for Hepatocellular Carcinoma

Background/Objectives: Treatment allocation for hepatocellular carcinoma (HCC) is a highly complex task that requires multidisciplinary tumor board (MDT) input; however, its accessibility and reliability can vary across the healthcare system. Large language models (LLMs) have emerged as potential clinical decision-maki...

C. Grapa, T. Mocan, D. Leucuța et al. · 0 citations
Open access Sep 2026

Large Language Models as Decision Support in Breast Cancer Management: A Comparison with Multidisciplinary Team Decision-Making

Background/Objectives: Breast cancer management relies on multidisciplinary team (MDT) decisions that integrate clinical, radiological, pathological, and patient-related factors. Large language models (LLMs) may support such decisions, but evidence based on real-world cases remains limited. Methods: We evaluated the ag...

G. Dindelegan, Noé Yoshi François Poupel, George Ionuț Golea et al. · 0 citations
#artificial intelligence Review Oct 2026

AI Consilium: a Benchmark of LLM Clinical Multidisciplinary Second Opinions

Large language models (LLMs) can generate differential diagnoses for complex cases, but a useful second opinion must also be factually grounded, clinically coherent, actionable, and safe. We developed a multidimensional benchmark using 300 diagnostically resolved NEJM--MedQA cases: 110 for physician reference ratings a...

A. Bokov, I. Panin, N. Nechaev et al. · 0 citations
Open access Sep 2026

A structured exploratory multidisciplinary evaluation of four large language models responding to frequently asked patient questions about pregabalin

Significant limitations persist in safety warning compliance, response comprehensiveness, and hallucination rate for selected basic pharmaceutical questions, and significant limitations persist in safety warning compliance, response comprehensiveness, and hallucination rate.

Y. L. T. Bayala, I. A. Tinni, Thierry Boris Wend-Yam Yaméogo et al. · 0 citations
Review Open access Oct 2026

Performance of large language models in pediatric infectious disease education: a multi-condition comparative study.

BACKGROUND Evaluating how appropriate current large language models (LLMs) respond to common pediatric infectious disease (PID) questions is important for understanding their potential role in education and clinical support. Therefore, we evaluated and compared the clinical quality of answers produced by different LLMs...

K. Aykac, Osman Cubuk, O. Demir et al. · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.