Large language models for invasive urodynamic interpretation: a blinded comparison with experienced urologists

Abstract

Abstract Background and objective Large language models (LLMs) are increasingly explored as clinical decision-support tools, but their performance in invasive urodynamic interpretation remains poorly characterized. This study compared the diagnostic agreement of four LLM configurations with two blinded experienced urologists in invasive urodynamic interpretation. Methods This retrospective study analyzed 113 urodynamic studies (UDS) (62 female, 51 male). Two experienced urologists independently and blindly evaluated each anonymized PDF report. The same reports were analyzed by GPT-4o and Claude Sonnet 4 using two prompting strategies: rule-based (predefined diagnostic criteria incorporating ICS recommendations and operational thresholds) and intuitive (holistic clinical reasoning), yielding four analysis arms across six diagnoses: bladder outlet obstruction (BOO), detrusor underactivity (DU), hypocompliant bladder, hypercompliant bladder, atonic bladder, and normal urodynamic study. Agreement was quantified using Cohen’s kappa and Pearson correlation. Results Interobserver agreement between urologists was substantial to almost perfect (mean κ = 0.727), highest for DU (κ = 0.942). Extraction of the primary numerical urodynamic parameters was identical across all LLM configurations ( r = 1.000), although variability was observed for the derived indices (BOOI and BCI) with intuitive prompting. LLM–urologist agreement was highest for DU and BOO with the Claude-Intuitive strategy (κ up to 0.753 and 0.522, respectively), whereas rule-based prompting achieved the highest agreement for several threshold-defined diagnoses. Agreement increased substantially within consensus subsets in which both urologists independently reached the same conclusion (full-consensus, n = 67: DU κ = 0.802; diagnosis-specific, n = 110: DU κ = 0.769), indicating that much of the apparent disagreement was concentrated in cases lacking concordant urologist assessments. Conclusions Contemporary LLM configurations showed identical extraction of the primary numerical urodynamic parameters and achieved substantial agreement with experienced urologists for specific diagnostic categories, particularly detrusor underactivity, with lower agreement observed for bladder outlet obstruction. Agreement was higher in cases with concordant urologist assessments. These findings, based on six predefined diagnostic categories evaluated under two prompting strategies, support a potential role for LLMs as adjunctive decision-support tools for select urodynamic parameters, rather than as replacements for expert evaluation or as evidence of comprehensive urodynamic interpretive competence.