The Reproducibility Gap in Mammographic CAD: Quantifying Patient-Level Leakage Effects on CBIS-DDSM

Abstract

Studies with deep learning models for mammographic breast cancer classification routinely report AUC values exceeding 0.90 on public benchmarks; however, the validity of these results is rarely examined at the protocol level. A pervasive issue is patient-level data leakage, whereby images from the same patient appear in both training and test sets, leading to artificially inflated performance estimates. This study conducted a systematic comparison of three evaluation protocols on the CBIS-DDSM benchmark using EfficientNetB3 under strictly identical architecture, hyperparameters, and augmentation strategies. Protocol P1 applies image-level stratified k-fold cross-validation. Protocol P2 concatenates the official training and test splits before folding, introducing all 201 test patients into the training pool across 804 cumulative patient-fold overlaps—a severe leakage that affects the entirety of the test population. Protocol P3 applies strict patient-level splitting through StratifiedGroupKFold, exclusively on the official training split. The experiments demonstrate that P2 artificially inflates AUC by +6.6 percentage points (AUC = 0.896 vs. AUC = 0.830) relative to the corrected protocol, under otherwise identical conditions. Notably, all three protocols yield nearly identical cross-validation AUC (0.847–0.849 across protocols), confirming that standard validation metrics cannot detect this leakage—only the held-out test set reveals it. Protocol P3 achieves AUC = 0.830, sensitivity = 0.769, and specificity = 0.714, establishing a methodologically valid benchmark for mammographic mass classification on CBIS-DDSM. These findings advocate for explicit patient-level leakage verification as a standard reporting requirement in medical imaging benchmarks.