Comparative Analysis of Turkish AI Chatbot Responses on Pediatric Atopic Dermatitis: Reliability, Quality, Readability, and Safety

Abstract

Objective: To compare the clinical accuracy, content quality, reliability, readability, and safety profiles of three advanced large language models (GPT-5.1 Thinking, Gemini 3 Pro, and Claude 4.5 Sonnet) in responding to frequently asked questions about pediatric atopic dermatitis management. Materials and Methods: Sixty questions covering diagnosis, triggers, skin care, medical treatments, and complication management were submitted to each model in Turkish. Primary analyses were performed on T1 responses; a second session (T2) was conducted one week later to assess test-retest reliability. Responses were evaluated by three blinded experts using a 4-point clinical accuracy scale, the Global Quality Score (GQS), Modified DISCERN (mDISCERN), and validated Turkish readability indices. Safety was assessed using a two-tier harm classification. Results: GPT-5.1 Thinking demonstrated the highest clinical accuracy (98.3% comprehensive/correct responses), with superior GQS and mDISCERN scores (median 5 for both). Claude 4.5 Sonnet showed comparable performance. Gemini 3 Pro exhibited significantly lower accuracy (88.3%) and reliability scores (median mDISCERN score of 3), despite producing the most readable content. No major harm was identified across the models; however, moderate harm (omission of critical details) was significantly more frequent with Gemini 3 Pro (23.3%) than with GPT-5.1 Thinking (10.0%). All models demonstrated strong stability in accuracy scores between sessions. Conclusion: GPT-5.1 Thinking exhibited superior accuracy and reliability for pediatric atopic dermatitis queries. The observed readability-safety trade-off with Gemini 3 Pro underscores the need for clinician oversight when using AI-generated content for parent education.