Quality Assessment of a Large Language Model’s Answers to Frequently Asked Questions About Meniscus Surgery: An Observational Study

Abstract

To assess the adequacy and the need for adjustment of answers generated by a large language model to common patient questions about meniscus surgery. Ten patient-oriented questions in Portuguese (Brazilian), reflecting frequent pre- and postoperative concerns about meniscus surgery, were submitted to ChatGPT (OpenAI; insert model version and access date) in independent sessions without follow-up prompts. Answers were qualitatively evaluated by an expert knee surgeons using a framework adapted from prior orthopedic chatbot studies: (1) Excellent and (2) Good but needs adjustment (when relevant omissions or insufficient clinical specificity were identified). Eight out of 10 answers (80%) were rated as Excellent and two (20%) as Good but needing adjustment. The main issues requiring refinement were related to (a) variability in expected hospital stay/discharge and (b) weight-bearing and functional progression after meniscal repair, which are protocol- and technique-dependent. The evaluated model provided largely adequate introductory information for common questions about meniscus surgery, but questions involving postoperative logistics and rehabilitation required additional nuance. Large language models should be used as supportive educational tools and should not replace individualized counseling by knee surgeons.