Jul 2026· IEEE/ACM International Conference on Connected Health: Applications, Systems and Engineering Technologies· Vol abs/2607.09982· 1 citation· 28 references
Computer Science
TL;DR
An explicit multimodal routing framework for clinical prediction that enables interpretable, robust, and auditable reasoning across three EHR modalities and introduces inference-time route masking, which simulates missing modalities and reweights the remaining routes without retraining.
Abstract
Electronic health record (EHR) data are inherently multimodal, and leveraging multiple modalities can improve predictive performance. However, most existing approaches rely on deep fusion, which obscures how individual modalities contribute to predictions and limits the interpretability of multimodal reasoning. We propose an explicit multimodal routing framework for clinical prediction that enables interpretable, robust, and auditable reasoning across three EHR modalities: structured longitudinal variables (L), clinical notes (N), and chest X-rays (I). Our model constructs discrete unimodal, directional bimodal, and trimodal routes to capture both individual modality signals and asymmetric cross-modal interactions. To audit multimodal reasoning and assess robustness, we introduce inference-time route masking, which simulates missing modalities and reweights the remaining routes without retraining. We analyze changes in performance and routing weights under these scenarios to understand model decision-making. We evaluate our framework on multi-label phenotype prediction (K = 25) and binary ICU mortality prediction using trimodal patient stays from MIMIC-IV, revealing systematic differences in modality reliance across clinical condition groups. Overall, our framework offers a transparent, auditable, and practical approach to multimodal clinical prediction, providing interpretability, robustness, and insights into how different data sources drive model decisions.
This work proposes MultiSigBERT, a unified framework for multimodal sequential survival modeling in oncology based on path signature representations that achieves a concordance index of 0.743 on an independent test set, demonstrating the benefit of jointly modeling multimodal temporal dynamics together with patient-level geometric structure for survival prediction.
Paul Minchella, Stéphane Chrétien, Guillaume Metzler et al.· 0 citations
Experimental results show that the proposed GuardMLLM improves performance on tasks such as predicting patient mortality and ICU length of stay, and effectively alleviates overconfidence in LLM.
G. Fu, Xiaolong Xu, Haolong Xiang et al.· IEEE journal of biomedical a...· 0 citations
Multimodality survival analysis for nasopharyngeal carcinoma (NPC) holds great potential for improving prognosis prediction and clinical decision-making. However, it is challenged by structural and semantic misalignments across heterogeneous data. Structural misalignment arises from incomplete clinical records, where missing data introduce uncertainty in prediction. Semantic misalignment stems from the gap between structured modalities (e.g., clinical and radiomic features) and unstructured data such as 3-D magnetic resonance imaging (MRI), hindering effective feature integration. Existing methods often ignore missing data or compress multimodal information into scalar representations, failing to capture complex modality interactions and solve the problem of semantic misalignment. Furthermore, current completion techniques typically lack interpretability and overlook joint modeling of inter- and intra-sample correlations when dealing with structural misalignment, limiting their reliability in clinical settings. These issues are further exacerbated by over-parameterized models prone to overfitting in small-sample scenarios. To address these challenges, we propose LMCF, a large language model guided multimodal completion and fusion (LMCF) network tailored for survival analysis with incomplete data. LMCF consists of two core components: a lightweight dual-branch multimodality enhanced feature encoding (LDME) layer, which incorporates an interpretable multisource cross-modality completer (IMCC) for explainable reconstruction of missing data to resolve structural misalignment; and a large language model (LLM)-guided structure-semantic two-stream fusion (LSTF) layer, equipped with a quaternion convolution-based cross-domain adaptive attention fusioner (QCAAF) to effectively integrate features across modalities and mitigate semantic misalignment. Extensive experiments on the Cancer Genome Atlas (TCGA) and two proprietary NPC datasets [postradiation nasopharyngeal necrosis (PRNN) and nasopharyngeal carcinoma dataset (NCD)] from Sun Yat-sen University Cancer Center demonstrate LMCF’s superior performance in survival prediction and risk stratification, particularly under conditions of incomplete modalities and limited data resources.
Fen Ling, Haoming Zeng, Ming Li et al.· IEEE Transactions on Computa...· 0 citations
BERT-LER is presented, a BERT-style model for coded EHR timelines pretrained and fine-tuned from a de-identified EHR dataset of 75 million patients, that encodes laboratory test results as discrete tokens while retaining graded information through percentile-based binning, paired with Integrated Gradients for token-level attributions grounded in the input EHR sequence.
Jun Ni Du, Lukas Adamek, Maxim A Kryukov et al.· 0 citations
Incident risk prediction from longitudinal electronic health records (EHRs) is challenging because relevant signals are multimodal, weak in isolation, and distributed across irregular patient histories. We propose structured evidence routing, a router-predictor-reviewer workflow that separates full-record access from disease-specific assessment. The router organizes the complete pre-index EHR into a compact summary and targeted evidence slices; the predictor uses this evidence to form an evidence-linked risk assessment, which the reviewer critiques. For comparison with supervised EHRSHOT baselines, we pair the routed evidence summaries with a supervised classifier readout. Across five 1-year incident diagnosis tasks, our method reaches the AUROC range of established supervised EHRSHOT baselines and remains competitive on AUPRC, while exposing a patient-specific evidence trail. Internal pre-readout ablations further suggest that routing, laboratory evidence, task guidance, and review each contribute to performance.
Animesh Agarwal, Meysam Ghaffari, Nina Fatehi et al.· 0 citations
