Evaluating the generalisability of LiSep LSTM for early prediction of septic shock across US and european cohorts

Abstract

Sepsis remains a major public health concern and is associated with high mortality. Early detection and timely intervention are critical for improving outcomes, yet no standardised approach has been universally adopted. The LiSep LSTM model, a Long Short-Term Memory neural network developed using the MIMIC-III database, has demonstrated promising predictive performance but generalisability across regions has not yet been explored. This study evaluated the LiSep LSTM model in an independent European cohort from the AmsterdamUMC (AUMC) database. Model performance was assessed using internal, external, and pooled validation to examine the effects of distributional shift and data heterogeneity across cohorts. Permutation feature importance (PFI) was further applied to quantify each feature’s contribution to model performance across evaluation settings. When tested on the AUMC cohort, the model achieved an Area Under the Receiver Operating Characteristic Curve (AUROC) of 0.8851 (95% confidence interval [CI] 0.8612–0.909) with a median Hours Before Onset (HBO) of 47.0 hours (interquartile range [IQR]: 16.0–255.0), slightly outperforming its performance on MIMIC-III. External validation showed reduced calibration and recall, while pooled training improved AUROC and Brier score relative to external-only evaluation. Accuracy was moderate (0.60–0.77) and precision remained low (0.01–0.17), indicating a substantial false-positive burden. These findings suggest that pooled training enhances robustness and supports earlier identification of septic shock, but local calibration and strategies to manage false alarms remain necessary for reliable deployment.