Evaluating OpenEvidence Across Pediatric Subspecialties: A Guideline-Anchored Expert Review
Abstract
Objective
To evaluate the quality, safety, and citation characteristics of OpenEvidence across pediatric subspecialties.
Methods
Guideline-anchored clinical questions spanning high, moderate, and low evidence certainty across 6 pediatric subspecialties were submitted to OpenEvidence. Expert reviewer pairs independently scored responses using the validated Quality Analysis of Medical Artificial Intelligence (QAMAI) instrument (6–30 range, higher scores indicate better quality) and a modified Agency for Healthcare Research and Quality harm assessment (0–7 range, higher scores indicate more harm), followed by consensus reconciliation. Citation analysis examined overlap with guideline references, claim substantiation, and pediatric specificity. Response consistency was assessed across 5 repeated queries.
Results
Seventeen questions were evaluated. The median QAMAI score was 27/30 (IQR, 24–29; “very good” quality). Potential harm was low (median 0/7, range 0–3), though 6 of 17 responses carried nonzero harm potential. OpenEvidence cited the primary guideline in 15 of 17 questions, yet only 17.6% of responses showed strong citation overlap (≥75% shared references) with the corresponding guidelines. OpenEvidence’s citations reliably supported the claims they accompanied (88.2% with high substantiation), and 70.8% of reviewer assessments rated the evidence base as comparable in quality to that of the corresponding guidelines. In total, 35.3% of responses cited studies lacking pediatric specificity. Responses were clinically consistent across repeated queries (77.9% had no meaningful or only minor variation); no contradictory recommendations were observed.
Conclusions
OpenEvidence produced high-quality responses with low expert-rated potential for harm across multiple pediatric subspecialties. The pediatric specificity of cited evidence warrants attention, reflecting the broader challenge of applying artificial intelligence tools that are trained on adult-predominant literature to pediatric populations.
