It is shown that merely adding auxiliary modalities does not guarantee improvement on ICU mortality prediction over an EHR-only baseline, and implies that careful design of the fusion architecture and an appropriate evaluation in the clinical context are required.
Abstract
Autoregressive foundation models trained on tokenized electronic health records (EHRs) can support zero-shot clinical prediction, yet most operate on structured event codes alone, and do not incorporate multiple modalities in a principled way. We present a framework for conditioning such models on auxiliary clinical modalities, including ECG waveforms, chest X-ray images, and clinical notes, using modality-specific latent compression and gated cross-attention with temporal alignment. We investigate two key design choices: (1) how to compress long per-modality sequences (e.g., ECG time series) before they enter the multi-modal cross-attention. This feature may be essential to reduce compute overheads and may be beneficial for generalization; (2) how the choice of pretrained encoder for each modality impacts downstream performance. Through controlled ablations on MIMIC-IV, we show that the best latent-compression configurations outperforms both uncompressed cross-attention and mean pooling. Encoder choice has a clear within-modality effect, with stronger pretrained encoders consistently outperforming weaker alternatives. We further show that merely adding auxiliary modalities does not guarantee improvement on ICU mortality prediction over an EHR-only baseline. This implies that careful design of the fusion architecture and an appropriate evaluation in the clinical context are required.
Autoregressive foundation models for electronic health records (EHRs) typically inherit pretraining methods from language modeling, where patient trajectories are concatenated into a single token stream and windows are sampled from that stream. In EHR data, this choice is consequential: windows may mix multiple patients, and patients with longer records contribute more optimization updates, potentially introducing bias. We propose Patient Sampling, a pretraining sequence-construction method that allows us to control how training signal is distributed across patients. We compare this method to the standard approach, which we refer to as Global Stream. We show that stochastic Patient Sampling with controllable weighting improves performance on real-world EHR data. Across downstream clinical tasks on MIMIC-IV v2.2 and v3.1, Patient Sampling improves Macro AUROC and AUPRC over the Global Stream baseline. These results identify training and validation sequence construction as important and underexplored design choices for autoregressive EHR foundation models.
Joshua Placidi, Yuxuan Liu, Jinpei Han et al.· arXiv.org· 0 citations
LLM4EHR, a new clinical foundation model trained on ICU EHR data, is proposed and it is found that learnt EHR TS embeddings from LLM4EHR improve performance on various downstream clinical tasks with competitive performance.
Jingteng Li, A. Capstick, Louise Rigny et al.· arXiv.org· 0 citations
Results indicate that a single serialization-based paradigm, without bespoke fusion architectures, is sufficient for multimodal clinical prediction - substantially reducing system complexity while matching or exceeding specialized designs.
A. Ravichandran, Bilgin Osmandoja, Klemens Budde et al.· arXiv.org· 0 citations
Predicting 30-day hospital readmission is essential for assessing patient stability and optimizing healthcare resources. As clinical risk evolves with the accumulation of evidence during hospitalization, capturing these dynamic trajectories is essential. However, many existing approaches compress the complex longitudinal history into fixed representations, often losing the granular, day-level clinical signals that reflect a patient's evolving physiological state. To address this, we propose Mr.Dec (Multimodal Readmission-risk prediction Decoder), which models each admission as a natural chronological sequence of daily multimodal events. By leveraging a Transformer Decoder, Mr.Dec integrates daily Electronic Health Record(EHR) updates and intermittent Chest X-ray(CXR) findings in a time-aligned stream, reflecting the actual clinical workflow. To ensure robustness, we utilize Disease-Specific Supervised Contrastive Learning as an auxiliary regularization to induce a diagnosis-aware structure in the latent space. Evaluations on the MIMIC-IV and MIMIC-CXR datasets show that Mr.Dec achieves state-of-the-art performance by preserving the integrity of the clinical sequence. Furthermore, our model identifies"Critical Days"within an admission, providing actionable and clinically grounded interpretations for real-time risk stratification. Code is available at: https://github.com/yejix-ai/MR.DEC
BERT-LER is presented, a BERT-style model for coded EHR timelines pretrained and fine-tuned from a de-identified EHR dataset of 75 million patients, that encodes laboratory test results as discrete tokens while retaining graded information through percentile-based binning, paired with Integrated Gradients for token-level attributions grounded in the input EHR sequence.
Jun Ni Du, Lukas Adamek, Maxim A Kryukov et al.· 0 citations
Foundation models for physiological signals have shown promise for health monitoring, with recent work exploring multi-modal approaches that leverage complementary information across modalities such as PPG and ECG. However, existing methods typically rely on contrastive objectives or reconstruction losses that operate in the input space, making them susceptible to sensor noise inherent in physiological recordings. We propose a multi-modal Joint-Embedding Predictive Architecture for PPG and ECG. Building on LeJEPA's theoretically grounded framework, our approach performs predictions entirely in a unified latent space, avoiding the need to model input-level noise. Separate encoders map each modality to a shared embedding space regularized toward an isotropic Gaussian -- a distribution provably optimal for downstream task performance. Stochastic modality dropout during training ensures each encoder learns complete representations, enabling inference and downstream fine-tuning with only a single modality despite training on paired PPG-ECG data. This flexibility is critical for real-world deployment where ECG is unavailable but its richer cardiac information can still inform PPG representations learned during pretraining. We evaluate our approach on a wide range of PPG downstream physiological assessment tasks, such as heart rate estimation, atrial fibrillation detection, and blood pressure estimation, demonstrating substantial improvements over existing baselines. Furthermore, we show that the model learns robust cross-modal alignment without relying on contrastive learning. Finally, we demonstrate that performance remains robust even with lower-quality signals, a common scenario for real-world PPG recordings.
Zhaoliang Chen, Saurabh Kataria, Minxiao Wang et al.· Proceedings of the 32nd ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
