Artificial intelligence meets pediatric orthopedics: A comparative analysis of ChatGPT-4o, Gemini 2.0, and Claude 3.5 in detecting supracondylar humeral fractures
TL;DR
Current LLMs demonstrate limited capability as independent diagnostic tools for pediatric supracondylar humeral fractures and require specialized pediatric training before clinical implementation, however, their potential as assistive tools for triage and assessment warrants further development of pediatric-specific models.
Abstract
Background Supracondylar humeral fractures constitute 10–16% of pediatric skeletal injuries, requiring timely diagnosis to prevent neurovascular complications. Developmental variations in pediatric bone structures pose diagnostic challenges for clinicians. This study evaluated three next-generation large language models (LLMs) (ChatGPT-4o, Gemini 2.0, Claude 3.5) for detecting pediatric supracondylar humeral fractures and their classification according to the Gartland system. Methods This retrospective observational study included 300 pediatric patients (150 with supracondylar humeral fractures confirmed by expert consensus, 150 without fractures) aged 2–10 years presenting to the Emergency Department of the Bilkent City Hospital (October 2022-January 2025). Two-view elbow radiographs were presented to each LLM three times on different days. Diagnostic accuracy was evaluated using overall accuracy (all three responses correct), strict accuracy (≥2 correct responses), and ideal accuracy (≥1 correct response). Response consistency was assessed using Fleiss’ Kappa coefficient. Fractures were classified according to modified Gartland criteria. Results Gemini 2.0 demonstrated highest sensitivity (68.4%) followed by Claude 3.5 (58.7%) and ChatGPT-4o (19.3%) for fracture detection (p < 0.001). Ideal accuracy rates were 83.3%, 78.7%, and 27.3% respectively. Although ideal accuracy rates exceeded 91% in non-fracture cases, specificity remained low (33.1–36.0%), indicating a high rate of false-positive classifications. Response consistency was very good for ChatGPT-4o (κ = 0.69) and Gemini 2.0 (κ = 0.61), good for Claude 3.5 (κ = 0.44). For Gartland classification, Gemini 2.0 achieved highest accuracy: Type I (83.3%), Type II (62.4%), Type III (68.7%). Conclusion Current LLMs demonstrate limited capability as independent diagnostic tools for pediatric supracondylar humeral fractures. Gemini 2.0’s 68.4% sensitivity indicates these technologies require specialized pediatric training before clinical implementation. However, their potential as assistive tools for triage and assessment warrants further development of pediatric-specific models.