Assessing the information quality of AI-generated patient educational materials for diabetes: a scoping review
TL;DR
Future research should prioritize validated, multilingual, and patient-centered evaluation tools that integrate conventional information-quality attributes with clinically relevant dimensions such as safety, actionability, personalization, transparency, empathy, and response efficiency.
Abstract
Objective To map the evidence on artificial intelligence (AI)-generated diabetes-related patient education materials and patient-facing health information, with particular attention to AI models, prompting approaches, evaluation methods, and information-quality outcomes. Methods This scoping review was conducted in accordance with the JBI methodology for scoping reviews and reported following the PRISMA-ScR checklist. The review was registered on the Open Science Framework (doi: 10.17605/OSF.IO/U4FAE) PubMed, Web of Science, Embase, Scopus, Cochrane CENTRAL, CNKI, WanFang Data, and SinoMed were searched from inception to May 1, 2026. Chinese- and English-language literature was searched. Two reviewers independently screened studies, charted data, and mapped reported outcomes to Wang and Strong's information quality framework. Outcomes not adequately represented by the framework were retained as additional dimensions. Descriptive statistics and narrative synthesis were used. Results Of 6,049 records identified, 24 studies from 11 countries or regions were included. All studies evaluated ChatGPT or another GPT-family model; 21 used zero-shot or direct prompting, three used role prompting, and two implemented retrieval-augmented generation. Eleven indicators were mapped to the information quality framework, with ease of understanding (n = 14), accuracy (n = 13), and believability (n = 9) assessed most frequently. Six additional outcomes were identified: clinical safety (n = 5), actionability (n = 3), response efficiency (n = 1), personalization (n = 1), transparency (n = 1), and empathy (n = 1). Most studies reported reading demands above those generally recommended for patient education, although findings varied by language, material type, and assessment method. Study-specific instruments were used in 17 studies (70.8%), whereas 10 (41.7%) used structured or established tools. Only six studies reported full source or model blinding, 10 reported quantitative inter-rater agreement, and three involved patients or members of the public. Conclusion Research on AI-generated diabetes education is expanding, but substantial heterogeneity in prompts, evaluators, tools, and outcome definitions limits comparison across studies. Future research should prioritize validated, multilingual, and patient-centered evaluation tools that integrate conventional information-quality attributes with clinically relevant dimensions such as safety, actionability, personalization, transparency, empathy, and response efficiency.