Large language models are used as synthetic clinical experts to supervise a variational-autoencoder-based approach that learns low-dimensional latent summaries of visit-level observations to make clinical knowledge available to representation learning and improve clinical faithfulness for longitudinal rare-disease data.
Abstract
Due to the limited amount of information, modeling longitudinal rare-disease data can benefit from integrating clinical knowledge. Yet, elicitation of expert knowledge and formalization for model fitting is challenging, in particular due to limited time of clinical experts. To nevertheless make domain knowledge accessible during model fitting, we use large language models (LLMs) as synthetic clinical experts to supervise a variational-autoencoder-based approach that learns low-dimensional latent summaries of visit-level observations. Specifically, LLMs are queried offline on textual descriptions of patient observations to obtain judgments, e.g., the suspected clinical category. To improve the variational autoencoder fit, we train a differentiable surrogate model on these judgments and augment the loss function to encourage reconstructions that preserve the clinical-label distribution of their corresponding input profile. In an application to longitudinal motor-function assessments from children with spinal muscular atrophy, we map visit-level clinical profiles to low-dimensional representations that are linked by a multivariate mixed-effects model. The synthetic expert loss discourages reconstructions that remain numerically close in data space but alter the clinical interpretation of the reconstructed motor function profile, such as by crossing a disease-type boundary. We thus reduced disagreement between original and reconstructed SMA type labels from about 11 to 7 percent. Furthermore, informing the latent representation by the synthetic expert improved prediction of motor function milestones compared with unsupervised latent representations and a data-level baseline. These results suggest that incorporating LLMs into model fitting can make clinical knowledge available to representation learning and improve clinical faithfulness for longitudinal rare-disease data.
Results indicate that a single serialization-based paradigm, without bespoke fusion architectures, is sufficient for multimodal clinical prediction - substantially reducing system complexity while matching or exceeding specialized designs.
A. Ravichandran, Bilgin Osmandoja, Klemens Budde et al.· arXiv.org· 0 citations
In clinical practice, patients often undergo multiple imaging examinations over successive visits, yielding longitudinal data. Modeling such temporal information is crucial for reliable assessment of disease progression and treatment response. However, despite the rapid advancement of multimodal large language models (MLLMs), longitudinal medical visual reasoning remains largely underexplored. To fill this gap, we propose LoMeVQA, a comprehensive benchmark consisting of 206K longitudinal visual question answering (VQA) pairs for temporal medical image analysis. LoMeVQA covers five tasks: progress classification, progress description, progress report generation, differential region grounding, and differential region description. To construct the dataset, we develop an automated pipeline that (1) organizes patient records chronologically, (2) extracts clinically meaningful entities via a medical knowledge graph, and (3) models their temporal evolution to guide large language models in generating high-quality longitudinal VQA pairs. Extensive evaluations demonstrate that both general-purpose and medical-domain MLLMs perform poorly on LoMeVQA, revealing substantial limitations in temporal reasoning. To address these limitations, we introduce MedLong-8B, which achieves state-of-the-art performance across all tasks. Beyond benchmarking, we conduct detailed analyses that uncover key failure modes and shed light on how to improve longitudinal medical visual reasoning. Our data is available at: https://github.com/pepperbubble/LoMeVQA
Zhilin Wu, Zhangkai Ni, Cheng Yang et al.· arXiv.org· 0 citations
Integrating heterogeneous clinical and molecular data for cancer prognosis remains challenging because their dimensionality, semantics and distributions differ across patients and cohorts. Here we present DeMoP, a language-model-guided mixture-of-experts framework that serializes structured patient profiles as natural-language sequences and learns adaptive prognostic representations from clinical variables, copy-number alterations, and gene descriptions. DeMoP combines a fine-tuned DeBERTa-v3-large encoder, attention-based token pooling, and a residual mixture-of-experts prediction head. In held-out tests from two independent pan-cancer cohorts, GENIE (63,090 patients) and TCGA (4,123 patients), DeMoP outperformed the conventional machine-learning and deep-learning baselines evaluated, achieving AUROCs of 0.939 and 0.805 and class-1 F1 scores of 0.72 in both cohorts. A GENIE-trained model transferred directly to TCGA with an overall class-1 F1 score of 0.62. Gene-level ablations recovered established cancer-associated genes and highlighted less-studied candidates. DeMoP provides a unified approach to heterogeneous biomedical data integration, cross-cohort outcome prediction, and model interpretation.
Chen Tang, Lei Yu, Qiwei Li et al.· bioRxiv· 0 citations
An engineering-oriented, end-to-end roadmap that structures the full lifecycle of clinical language model systems—from model design and domain adaptation to optimization and real-world evaluation is introduced.
Constructing interpretable disease models from longitudinal omics data is a central challenge in precision medicine. The goal is a low-dimensional representation in which a patient’s position encodes their molecular state and clinical severity, and along which disease progression can be read directly. Existing dimensionality reduction methods (e.g., UMAP, Variational Autoencoders) fall short of this goal: they optimize a single generic objective and are blind to clinical labels and to the temporal ordering of measurements. Consequently, trajectory inference is typically applied after the fact to an embedding that was never optimized to reveal progression, decoupling the representation from disease dynamics. Manifold learning offers a natural route to such representations, and we build on Locally Linear Embedding (LLE) to preserve the local geometry of the omics data (i.e., keeping molecularly similar patients close together in the low-dimensional space). Geometry alone, however, yields a space that is faithful to molecular similarity yet uninformative about clinical severity and progression. We therefore recast the problem as multi-task learning: MT-LLE jointly optimizes five objectives: geometric reconstruction, supervised organization by clinical stage, embedding and phenotype forecasting, and clustering. Because naively combining such heterogeneous objectives induces gradient conflicts that distort the molecular geometry, an embedded reinforcement learning agent dynamically schedules their weights during training, establishing global geometry before refining clinical boundaries. Across two independent Chronic Obstructive Pulmonary Disease (COPD) cohorts (SPIROMICS and COPDGene), MT-LLE deliberately relaxes exact geometric reconstruction, by a modest margin, in exchange for substantial gains in clinical structure. On held-out patients, a linear model reads disease severity (GOLD stage, 0–4) from the MT-LLE embedding 35–40% more accurately than from standard dimensionality reduction (0.53 vs. 0.38 F1-Macro). The gap is starker for progression: forecasting a patient’s next-visit severity from their trajectory reaches 0.38 F1-Macro, while unsupervised baselines sit near zero (0.06–0.09 F1-Macro), a temporal signal those methods fail to capture. To test whether the reinforcement learning agent earns its place, we compared it against a fixed schedule that imposes the same ordering of objectives but cannot adapt during training; the learned agent outperforms it by 13–20% across clinical metrics, showing the gains come from adapting the weights to how training unfolds, not merely from ordering the objectives correctly, and at no cost to geometric fidelity. Beyond these quantitative gains, the manifold supports complementary analyses that surface structure invisible to standard staging: static phenotyping isolates subjects with active molecular pathology despite preserved lung function; trajectory inference maps two mechanistically distinct progression axes (inflammatory fibrosis and pan-immune activation); and kinematic analysis of each patient’s speed and acceleration identifies subjects whose molecular trajectories accelerate ahead of detectable spirometric decline.
S. Hussein, I. Konigsberg, K. Kechris et al.· bioRxiv· 0 citations
A systematic study spanning five KG task formulations, three training paradigms, two KGs, and three base LLMs finds that at the task level, all paradigms improve over the non-finetuned baseline, but methods with comparable in-domain accuracy show substantially different knowledge transfer behavior.
Saksham Khatwani, He Cheng, Majid Afshar et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
