Benchmarking Multimodal Large Language Models for Cardiopulmonary Findings on Chest Radiographs: Sex-Stratified Discrimination and Operating Characteristics
TL;DR
Commercial MLLMs differ considerably in operating profiles, ranging from ultraconservative to aggressive detection, so that strong aggregate discrimination can mask sensitivity too low for reliable detection.
Abstract
Background/Objectives: To characterize the zero-shot diagnostic behavior of three commercial multimodal large language models (MLLMs) on cardiopulmonary chest radiograph findings and to assess sex-stratified performance differences. Methods: GPT-5.4, Claude Opus 4.5, and Gemini 2.5 Pro were evaluated in 4500 pathology-specific radiograph evaluations based on frontal chest radiographs from the publicly available CheXpert dataset. Three balanced cohorts of 1500 images each were constructed for cardiomegaly, pulmonary edema, and pleural effusion (375 per sex-by-label subgroup). All models received identical zero-shot prompts requesting binary classification. The primary outcome was area under the receiver operating characteristic curve (AUC-ROC) with 95% bootstrap confidence intervals. Secondary outcomes were sensitivity and specificity. Results: A total of 4500 pathology-specific radiograph evaluations were performed across the three cohorts (2250 male and 2250 female cohort entries; mean age 58.4 ± 18.0 years). GPT-5.4 achieved the highest discrimination (AUC-ROC 0.836–0.883) but showed very low sensitivity (0.043–0.424) with near-perfect specificity (0.977–0.997). Claude Opus 4.5 showed moderate discrimination (AUC-ROC 0.698–0.761) with balanced sensitivity (0.396–0.876) and specificity (0.461–0.863). Gemini 2.5 Pro showed moderate discrimination (AUC-ROC 0.745–0.770) but favored sensitivity (0.673–0.973) at the expense of specificity (0.241–0.804). Sex-stratified analyses showed consistently higher AUC point estimates in male patients for cardiomegaly and pulmonary edema, but smaller and less directional differences for pleural effusion. Conclusions: Commercial MLLMs differ considerably in operating profiles, ranging from ultraconservative to aggressive detection, so that strong aggregate discrimination can mask sensitivity too low for reliable detection. None of the evaluated models are currently suitable for autonomous chest radiograph interpretation. Sex-stratified differences were modest but non-uniform, supporting subgroup-aware reporting rather than reliance on pooled metrics alone.