Comparative evaluation of ChatGPT-4o and ChatGPT-5.1 for automated LUTS scoring from clinician-documented narratives

Abstract

Lower urinary tract symptoms (LUTS) are commonly quantified using validated instruments such as the International Prostate Symptom Score (IPSS) and the Overactive Bladder Symptom Score (OABSS). However, structured questionnaires are not consistently administered in routine outpatient practice, where clinicians frequently rely on narrative documentation. While generative large language models (LLMs) have demonstrated potential in clinical natural language processing, their performance stability across successive model generations for automated symptom scoring remains unclear. We conducted a head-to-head comparison of ChatGPT-4o and ChatGPT-5.1 to evaluate generational performance differences in automated LUTS scoring. This retrospective secondary analysis of a previously reported outpatient cohort evaluated the performance of ChatGPT-4o and ChatGPT-5.1 in estimating LUTS-related scores from clinician-documented narratives. Free-text symptom narratives documented from electronic medical records were entered directly into each model using identical prompts. Ground-truth IPSS and OABSS total scores were obtained from completed questionnaires. Predictive performance was assessed using mean absolute error (MAE), root mean squared error (RMSE), intraclass correlation coefficients (ICC), and Spearman’s correlation coefficients. Classification of clinically significant LUTS (IPSS ≥ 8) and overactive bladder (OAB; OABSS ≥ 3 with urgency score ≥ 2) was evaluated using accuracy, Cohen’s kappa (κ), and area under the receiver operating characteristic curve (AUC). The analysis included 91 patients from the original cohort. Among 91 patients (mean age 71.9 ± 10.3 years; 58.2% male), 49 completed IPSS and 78 completed OABSS. For IPSS prediction, MAE/RMSE were 4.88/6.34 for ChatGPT-4o and 4.92/7.01 for ChatGPT-5.1, with no significant differences between models. For OABSS prediction, MAE/RMSE were 1.88/2.76 and 2.03/2.93, respectively, again without significant differences. Agreement was moderate for IPSS (ICC 0.561–0.509) and higher for OABSS (ICC 0.704–0.658). Classification accuracy exceeded 0.85 for both outcomes, and ROC analysis demonstrated comparable discriminative performance across model versions. ChatGPT-4o and ChatGPT-5.1 demonstrated comparable and moderate performance in approximating LUTS-related symptom scores from unstructured clinical narratives. These findings suggest that generative AI models may serve as supportive tools for automated symptom quantification and indicate relative performance stability across model generations within real-world urological practice.