Responses of large language models to common questions on premature ovarian insufficiency: A comparative analysis.
TL;DR
GPT-5.5, Claude Opus-4.7, and DeepSeek-R1 demonstrated favorable accuracy and completeness in answering standardized POI questions; overall performance was similar despite deviations across several individual questions.
Abstract
Objective
Large language models are widely used in clinical care. Their reliability in specialized gynecological diseases such as premature ovarian insufficiency (POI) has not been fully verified. This study aimed to systematically evaluate and compare the accuracy and completeness of three mainstream large language models when answering standardized clinical questions about POI.
Methods
We developed 34 standardized questions covering five core domains of POI according to international clinical guidelines. Two senior obstetricians and gynecologists independently scored responses using a 5-point Likert scale. Inter-rater reliability was assessed using Cohen's kappa coefficient, intraclass correlation coefficient, and mean absolute error. Pairwise between-model score differences were computed and visualized using item-level heatmaps.
Results
All three models showed high and steady performance. Accuracy scores ranged from 4.59 to 4.84, and completeness scores ranged from 4.43 to 4.65. Inter-rater agreement was "good" to " excellent", with exact agreement between 91.18% and 97.06% and Cohen's kappa between 0.675 and 0.915. GPT-5.5 performed better in the area of clinical manifestations and treatment. DeepSeek-R1 showed greater stability, defined as maintaining consistently high scores with limited score fluctuation, in etiology- and counseling-related questions. Item-level heatmaps revealed that most individual-item differences were within ±0.5 points, whereas pairwise differences in overall mean scores fell within ±0.5, and some individual-item comparisons exceeded this range, with no obvious systematic bias.
Conclusions
GPT-5.5, Claude Opus-4.7, and DeepSeek-R1 demonstrated favorable accuracy and completeness in answering standardized POI questions; overall performance was similar despite deviations across several individual questions. Item-level difference heatmaps further confirmed the consistency and reliability of the three models. These models can serve as reliable auxiliary tools for clinical information retrieval and patient education under professional supervision, meaning that all content generated by artificial intelligence must be checked by qualified clinicians before clinical use.