Unmasking clinical shortcuts: a latent-space audit of dense convolutional networks for chest radiography

Abstract

Deep convolutional neural networks (CNNs) in medical imaging can achieve acceptable global discrimination while still relying on shortcut-sensitive signals that undermine reliability in clinically important subgroups. In chest radiography, non-pathological factors such as image acquisition view, support devices, and surgical hardware may correlate with disease labels and influence model behavior in ways that are not visible through aggregate metrics alone. This study presents a latent-space audit of DenseNet-121 and ResNet-50 models trained on the CheXpert dataset for binary pneumothorax detection. We apply a Slicing Discovery Method (SDM) to identify hidden regions of model failure. For each trained model, penultimate-layer embeddings were extracted from the validation set and compressed into 128-dimensional supervised slice representations. Gaussian Mixture Models were then fitted separately within positive and negative cases, with the number of clusters selected using Bayesian Information Criterion. Discovered slices were ranked using Brier score and calibration gap to identify regions of poor probabilistic reliability. Post hoc metadata analysis was used to characterize high-error slices by support-device prevalence and image acquisition view. DenseNet-121 achieved a validation AUROC of 0.7883 and global Brier score of 0.0671, while ResNet-50 achieved a validation AUROC of 0.7723 and global Brier score of 0.0702. Despite these comparable global metrics, both architectures exhibited severe localized reliability failures. The worst DenseNet-121 slice contained 417 confirmed positive pneumothorax cases but had a mean predicted probability of only 0.0402, yielding a Brier score of 0.9218 and calibration gap of 95.98%. ResNet-50 showed a similar but less severe worst-slice failure, with a peak Brier score of 0.8779. View-position analysis revealed that the most severe positive failure slices were dominated by portable anteroposterior views and produced high false-negative rates, including 100% falsenegative rates in the worst positive slices for both architectures. Support devices were highly prevalent in these failure slices, although not enriched above the already high validation-set baseline. These findings demonstrate that global AUROC can obscure clinically meaningful reliability collapse in latent subgroups. DenseNet-121 showed slightly stronger global discrimination but greater worst-slice vulnerability than ResNet-50 in this audit, suggesting that architectural design may influence the structure of shortcut-consistent failure modes. Latent-space auditing with SDM provides a practical pre-deployment framework for identifying hidden failure modes and improving the safety evaluation of medical imaging AI systems.