Evaluation of Turkish artificial intelligence-generated responses about osteoporosis in terms of accuracy, completeness, and readability

Abstract

Purpose: Artificial intelligence (AI) chatbots are increasingly used in patient education. This study evaluated the responses of AI chatbots to frequently asked questions about osteoporosis in Turkish in terms of accuracy, completeness, and readability. To our knowledge, this is the first study to evaluate AI-generated responses to osteoporosis-related questions in Turkish.Materials and methods: Thirty-one of the most common frequently asked questions related to osteoporosis were subcategorized into general information, diagnosis, and treatment. The questions were submitted to the publicly accessible platforms ChatGPT-5.3, Gemini, Copilot, and Perplexity. Two physicians evaluated accuracy (six-point Likert scale) and completeness (three-point Likert scale) based on the 2025 Guideline for the Diagnosis and Treatment of Osteoporosis and Metabolic Bone Diseases. Readability was assessed using the Ateşman Readability Formula.Results: No significant difference was found among the chatbots for accuracy. Accuracy rates ranged 80.5%-82.2% overall, 82.3%-85.2% general information, 58.3%-66.7% diagnosis, and 77.2%-81.7% treatment. Significant differences were found in completeness overall and in the general information (p=0.006, p=0.029; respectively). In pairwise comparisons, ChatGPT-5.3 showed significantly higher completeness than Perplexity (p=0.001, p=0.006; respectively). Completeness rates ranged 47%-64.3% overall, 46%-68.3% general information, 33.3%-50% diagnosis, and 48.3%-60.3% treatment. Readability differed significantly among the chatbots (p<0.001). Based on mean scores, the ranking was ChatGPT-5.3>Gemini=Copilot>Perplexity. ChatGPT-5.3, Gemini, and Copilot were easy, whereas Perplexity was moderately difficult.Conclusion: All chatbots showed similar accuracy, but ChatGPT-5.3 performed better in completeness and readability. Chatbots may be useful in Turkish patient education. However, their standalone use in osteoporosis management without physician supervision is not appropriate.