LLM4EHR, a new clinical foundation model trained on ICU EHR data, is proposed and it is found that learnt EHR TS embeddings from LLM4EHR improve performance on various downstream clinical tasks with competitive performance.
Abstract
Recent research in clinical machine learning, focusing on outcome predictions in intensive care unit (ICU), has shifted from bespoke supervised models to foundation models, utilising modern representation learning methods. Here, foundation models are pre-trained on mixtures of complex clinical data modalities, useful for various downstream tasks. Existing works often utilise Electronic Health Records (EHR) to provide rich and diverse patient observations to train clinical foundation models. However, existing methods do not sufficiently explore the shared temporal structures between clinical events and time series (TS) observations recorded in EHRs. This limitation potentially leads to less robust and adaptive clinical foundation models, resulting in reduced performance on downstream tasks. To fully exploit this temporal structure, we propose LLM4EHR, a new clinical foundation model trained on ICU EHR data. Combining domain adapted large language models with a transformer TS encoder, we pre-trained LLM4EHR by temporally aligning the EHR events and TS. For this, we propose a regularised contrastive objective to learn robust EHR TS representations conditioned on EHR event embeddings produced by the domain adapted LLM. Supported by an ablation study, we find that learnt EHR TS embeddings from LLM4EHR improve performance on various downstream clinical tasks with competitive performance. Further, we empirically demonstrate that LLM4EHR learns transferable clinical TS embeddings that can be deployed to new cohorts via k-shot adaptation. These findings provide a step towards building more generalisable and performant clinical foundation models.
It is shown that merely adding auxiliary modalities does not guarantee improvement on ICU mortality prediction over an EHR-only baseline, and implies that careful design of the fusion architecture and an appropriate evaluation in the clinical context are required.
Yuxuan Liu, Joshua Placidi, Jinpei Han et al.· arXiv.org· 0 citations
Results indicate that a single serialization-based paradigm, without bespoke fusion architectures, is sufficient for multimodal clinical prediction - substantially reducing system complexity while matching or exceeding specialized designs.
A. Ravichandran, Bilgin Osmandoja, Klemens Budde et al.· arXiv.org· 0 citations
BERT-LER is presented, a BERT-style model for coded EHR timelines pretrained and fine-tuned from a de-identified EHR dataset of 75 million patients, that encodes laboratory test results as discrete tokens while retaining graded information through percentile-based binning, paired with Integrated Gradients for token-level attributions grounded in the input EHR sequence.
Jun Ni Du, Lukas Adamek, Maxim A Kryukov et al.· 0 citations
Autoregressive foundation models for electronic health records (EHRs) typically inherit pretraining methods from language modeling, where patient trajectories are concatenated into a single token stream and windows are sampled from that stream. In EHR data, this choice is consequential: windows may mix multiple patients, and patients with longer records contribute more optimization updates, potentially introducing bias. We propose Patient Sampling, a pretraining sequence-construction method that allows us to control how training signal is distributed across patients. We compare this method to the standard approach, which we refer to as Global Stream. We show that stochastic Patient Sampling with controllable weighting improves performance on real-world EHR data. Across downstream clinical tasks on MIMIC-IV v2.2 and v3.1, Patient Sampling improves Macro AUROC and AUPRC over the Global Stream baseline. These results identify training and validation sequence construction as important and underexplored design choices for autoregressive EHR foundation models.
Joshua Placidi, Yuxuan Liu, Jinpei Han et al.· arXiv.org· 0 citations
The early prediction of chronic illnesses such as diabetes, hypertension, and heart failure is vital for improving patient outcomes and reducing healthcare costs. However, Electronic Health Records (EHRs) present challenges due to their heterogeneous nature, containing a mix of structured (e.g., laboratory values, vital signs), unstructured (e.g., clinical notes), and temporal data (e.g., patient history). This paper proposes a novel deep ensemble learning framework that integrates Temporal Convolutional Networks (TCNs) and attention-guided Long Short-Term Memory (LSTM) networks for robust chronic disease prediction. The TCN module captures long-range temporal patterns and clinical trends across patient timelines, while the attention-enhanced LSTM module focuses on relevant time-steps and feature segments critical for diagnosis. The ensemble approach combines these deep representations to improve classification accuracy, reduce false positives, and provide resilience against data irregularities. Experiments conducted on benchmark multi-modal EHR datasets demonstrate the model’s superiority over traditional RNN, LSTM, and CNN-based methods. The proposed architecture not only enhances prediction performance but also contributes toward explainable and proactive healthcare decision-making.
T. Thamaraiselvan, K. Saravanan, S. Nithyanandam· Journal of Intelligent Decis...· 0 citations
Background: Machine learning models leveraging electronic health records (EHRs) can support earlier detection of sepsis in intensive care units (ICUs). However, their clinical utility depends on reproducibility across institutions and patient populations. Building on a published pipeline from the Children's Hospital of Philadelphia (CHOP), this study examines how a neonatal sepsis prediction framework performs and can be adapted to a range of intensive care environments, paediatric, cardiac, and neonatal, at Great Ormond Street Hospital (GOSH). Methods: We extracted de-identified ICU EHR data from GOSH and applied feature derivation, unit harmonisation, and temporal sampling to align with the CHOP dataset used by Masino et al. (2019). Seven classifiers were first evaluated using CHOP-trained weights to characterise cross-domain behaviour and then retrained on local data to assess recoverability and site-specific adaptation. Model discrimination was summarised by AUC and F1, and learning curves were used to explore sample efficiency and bias-variance dynamics. Results: Models achieved strong discrimination on the CHOP neonatal cohort but demonstrated reduced performance when transferred to the mixed GOSH ICU population, reflecting anticipated domain and population shift. Retraining on GOSH data restored discrimination (AUC range 0.69-0.86), with Gradient Boosting (AUC 0.86 vs AUC 0.87 at CHOP) and KNN (AUC 0.80 vs AUC 0.79 at CHOP) models performing comparably to their CHOP benchmarks. DeLong's test confirmed statistically significant gains across all classifiers (p < 0.001). Conclusion: ICU cohort and baseline demographic differences between CHOP and GOSH introduced domain shift that limited direct model transfer. Elements of the original preprocessing pipeline could not be reproduced, further constraining transportability. Yet, retraining on local data restored high discrimination, showing that the modelling framework remains robust when re-estimated in new settings. These results highlight local adaptation as a practical route to recover performance and support safe, generalisable deployment of clinical prediction models in mixed clinical environments.
S. Champeaux, J. Booth, A. Brown et al.· medRxiv· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
