Skip to content
Review Open access

Evaluation of ChatGPT-Generated Simplifications of Shoulder MRI Reports: Assessments by Orthopedic Surgeons and Non-Healthcare Volunteers

Nov 2026 · Diagnostics · 0 citations · 20 references

Abstract

Background/Objectives: To evaluate ChatGPT-generated (GPT-5) simplifications of shoulder MRI reports from the perspectives of orthopedic surgeons and non-healthcare volunteers. Methods: In this single-center retrospective study, 100 de-identified shoulder MRI reports issued between June 2024 and June 2025 were simplified with ChatGPT in February 2026 using a standardized prompt. Between 10 March and 10 April 2026, three orthopedic surgeons independently rated each simplified text against the original report for accuracy, completeness, understandability, and misleading potential on a 5-point Likert scale (higher scores indicating better ratings). In a second arm, 100 adult non-healthcare volunteers each reviewed one simplified report and completed a six-item questionnaire on perceived clarity, comprehensibility, usefulness, and appropriateness. Results: Median expert scores were 5.0 in all domains, and all three raters assigned scores ≥ 4 in 86–98% of reports. Gwet’s AC2 ranged from 0.93 to 0.98, whereas Krippendorff’s α ranged from −0.02 to 0.05. Among volunteers, agreement ranged from 47% for understanding report terms to 80% for the appropriateness of the text. Health literacy correlated positively with most items (r = 0.329–0.429, p ≤ 0.001), and age correlated negatively with understanding report terms (r = −0.352, p < 0.001). Conclusions: The simplified reports were rated favorably by orthopedic surgeons and were perceived as useful by non-healthcare volunteers. However, the study was limited by its single-center design, the use of a single model and prompt, the absence of a comparison with the original reports in the volunteer arm, and the reliance on self-reported rather than objectively tested comprehension; therefore, it does not demonstrate improved patient comprehension or clinical safety. Such texts should supplement, not replace, the original report and physician consultation.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.