Generative AI chatbots as sources of phimosis-related consumer health information: a cross-sectional comparison of quality, source transparency, and readability

Abstract

Publicly accessible generative artificial intelligence chatbot products are increasingly used for sensitive health questions, but the quality, source transparency, and readability of phimosis-related information remain uncertain. We conducted an internet-based cross-sectional comparative study on 1 July 2026. 15 of 25 Google Trends candidate terms were submitted unchanged to ChatGPT (displayed as GPT-5.5), Microsoft Copilot, Gemini (displayed as 3.5 Flash), and Perplexity, yielding 60 responses. Two practising urologists holding senior professional titles, each with more than 10 years of clinical experience, independently assessed responses using DISCERN, Ensuring Quality Information for Patients (EQIP), the Global Quality Score (GQS), and Journal of the American Medical Association (JAMA) benchmark criteria. Inter-rater reliability was assessed using ICC (2,1) for DISCERN and EQIP and quadratic-weighted Cohen’s kappa for JAMA and GQS. Because the same 15 queries were evaluated across all four products, quality outcomes were analysed as matched data using Friedman tests with Kendall’s W and Holm-adjusted paired Wilcoxon signed-rank comparisons. Readability was assessed with six established indices. Inter-rater reliability was good to almost perfect (coefficients 0.866–0.882). After Holm adjustment across the four omnibus tests, differences remained for DISCERN ( p  = 0.007; W  = 0.301), EQIP ( p  < 0.001; W  = 0.587), JAMA ( p  < 0.001; W  = 0.953), and GQS ( p  = 0.049; W  = 0.174). For DISCERN, Copilot scored higher than ChatGPT and Gemini, and Perplexity scored higher than Gemini. For EQIP and JAMA, Copilot and Perplexity scored higher than ChatGPT and Gemini. No GQS pairwise comparison remained significant after Holm correction. None of the 15 responses from any product met any individual prespecified readability threshold. A sensitivity analysis excluding the broad standalone prompts ‘circumcision’ and ‘balanitis’ preserved the DISCERN, EQIP, and JAMA findings but not the GQS omnibus difference. In this purposively derived 15-query, single-date snapshot, chatbot products differed in selected measures of information quality and source transparency, while readability remained consistently above the prespecified sixth-grade target. These findings are conditional on the sampled queries and outputs and should not be interpreted as stable platform-wide estimates of clinical accuracy or safety.