Performance of large language models and pediatricians in complex pediatric scenarios: a prospective, randomized comparative study of clinical task accuracy and decision support

Abstract

To compare response accuracy between contemporary large language models (LLMs) and pediatricians across difficulty-stratified pediatric clinical vignettes, characterize LLM response times, and quantify the effect of AI-generated outputs on pediatrician decisions. In a prospective, randomized, assessor-blinded, vignette-based comparative study, 120 pediatric questions (four difficulty levels) were administered to five LLMs (ChatGPT-5, Gemini Pro 2.5, Claude Opus 4.1, Super Grok 4, DeepSeek V3) and to 30 pediatricians and to 30 pediatricians randomized 1:1 to the Control group (unassisted) or the AI-Assisted group. Models were queried in separate sessions with tools/browsing disabled using an identical prompt template. In the AI-Assisted group, pediatricians recorded an initial answer and confidence (1–10), reviewed five anonymized AI outputs presented in random order, and then submitted a final answer. A blinded adjudicator scored all responses against a prespecified answer key; p-values were adjusted via Holm–Bonferroni. Mixed-effects logistic regression was additionally performed to account for clustering of repeated responses within pediatricians and items. Response accuracy was 78.50% in the Control group, 83.28% in the AI-Assisted group after exposure to AI-generated outputs ( p = 0.0003; Cohen’s d = 0.52), and 93.00% across the five LLMs evaluated independently (individual-model range, 87.50%–99.17%). In mixed-effects logistic regression accounting for repeated responses within pediatricians and items, Control group pediatricians had lower odds of a correct response than AI-Assisted pediatricians (OR = 0.80, 95% CI 0.69–0.94; p = 0.007). Response times differed across models ( p < 0.001), and performance varied according to item difficulty and clinical domain. In this standardized, vignette-based evaluation, contemporary LLMs achieved higher response accuracy than the participating pediatricians, and AI assistance improved physicians’ response accuracy. These findings should not be interpreted as evidence of superiority in real-world clinical practice; rather, they support the potential use of LLMs as clinician-supervised decision-support tools, with safeguards against automation bias. ClinicalTrials.gov (NCT07179861) first submitted 11 September 2025, first posted 18 September 2025. Retrospectively registered. ClinicalTrials.gov (NCT07179861); first submitted 11 September 2025, first posted 18 September 2025. Retrospectively registered.