MedPMC, an automated, continuously updatable framework that transforms permissively licensed literature into high-fidelity infrastructure for medical multimodal models, is introduced and publicly release the framework, corpus, benchmarks, and pretrained models.
Abstract
Medicine is inherently multimodal, requiring clinicians to synthesize information across diverse data streams. Yet the development of multimodal foundation models is constrained by limited access to large-scale, high-quality clinical data. Although PubMed Central (PMC) offers a complementary source of expert-authored image-text data, existing PMC-derived resources remain limited in fidelity, reproducibility, and clinical validation. We introduce MedPMC, an automated, continuously updatable framework that transforms permissively licensed literature into high-fidelity infrastructure for medical multimodal models. Applied to 6.1 million PMC articles, MedPMC curated 11 million medical image-text pairs. Component evaluations showed strong performance for initial screening (F1 = 93.2), multi-panel figure detection (F1 = 96.5), figure separation (mAP = 89.8), caption separation and alignment (F1 = 81.4; ROUGE-L = 85.3), and medical figure classification (F1 = 96.5). Manual review by five annotators, three with medical training, found 95.3% of MedPMC images medically relevant, versus 19.7% in a prior PMC-derived dataset. Across 26 benchmarks spanning 11 specialties, a MedPMC-trained CLIP-style model improved average zero-shot AUC by 7.1 percentage points over the strongest architecture-matched biomedical CLIP baseline despite using fewer than half as many image-text pairs. As the vision encoder in a multimodal large language model, it improved medical visual question-answering by 1.9 and 16.9 percentage points across two benchmarks. In 10,524 Yale New Haven Health System dermatology photographs, it improved morphology-to-image retrieval Recall@5 by 11.7 percentage points. These findings show that high-fidelity literature curation strengthens medical multimodal foundation models across benchmark and clinical settings. We publicly release the framework, corpus, benchmarks, and pretrained models.
This work introduces MedReaMM, a benchmark specifically designed to evaluate models'ability to synthesize heterogeneous clinical evidence consisting of detailed patient histories alongside multiple medical images into accurate differential diagnoses under a complete-information paradigm.
Lai Wei, Yu-Chao Chen, Zhenbiao Cao et al.· 0 citations
Recent advancements in foundation models (FMs) have catalyzed a paradigm shift in medical image analysis. Unlike traditional task-specific artificial intelligence (AI) models, FMs leverage large-scale datasets to learn generalized representations that can be adapted to downstream clinical applications. Despite the rapid proliferation of FM research in medical imaging, there is a lack of unified synthesis that systematically maps the evolution of architectures, training paradigms, and clinical applications across modalities. To address this gap, this review provides a comprehensive and structured synthesis of FMs in medical image analysis by systematically organizing studies into two primary categories: vision-only foundation models (VFMs) and vision-language foundation models (VLFMs), based on their architectural foundations, training strategies, and downstream clinical tasks. A quantitative analysis was conducted on both VFMs and VLFMs to characterize temporal trends in dataset utilization and application domains, along with pooled performance and subgroup analyses. We also critically discuss persistent challenges, including cross-domain generalization, computational scalability, FM evaluation, fairness, and deployment. Finally, we identify key future research directions aimed at enhancing the robustness, interpretability, and clinical integration of FMs, thereby accelerating their translation into real-world medical practice.
P. Rajendran, M. Safari, Wen-Feng He et al.· Medical Image Analysis· 0 citations
In clinical practice, patients often undergo multiple imaging examinations over successive visits, yielding longitudinal data. Modeling such temporal information is crucial for reliable assessment of disease progression and treatment response. However, despite the rapid advancement of multimodal large language models (MLLMs), longitudinal medical visual reasoning remains largely underexplored. To fill this gap, we propose LoMeVQA, a comprehensive benchmark consisting of 206K longitudinal visual question answering (VQA) pairs for temporal medical image analysis. LoMeVQA covers five tasks: progress classification, progress description, progress report generation, differential region grounding, and differential region description. To construct the dataset, we develop an automated pipeline that (1) organizes patient records chronologically, (2) extracts clinically meaningful entities via a medical knowledge graph, and (3) models their temporal evolution to guide large language models in generating high-quality longitudinal VQA pairs. Extensive evaluations demonstrate that both general-purpose and medical-domain MLLMs perform poorly on LoMeVQA, revealing substantial limitations in temporal reasoning. To address these limitations, we introduce MedLong-8B, which achieves state-of-the-art performance across all tasks. Beyond benchmarking, we conduct detailed analyses that uncover key failure modes and shed light on how to improve longitudinal medical visual reasoning. Our data is available at: https://github.com/pepperbubble/LoMeVQA
Zhilin Wu, Zhangkai Ni, Cheng Yang et al.· arXiv.org· 0 citations
Artificial intelligence (AI)-based medical imaging diagnosis has demonstrated remarkable performance across multiple clinical domains, with deep learning models frequently reporting diagnostic accuracy, sensitivity, and specificity exceeding 90% under controlled experimental conditions. However, translating these results into clinically reliable, regulatory-compliant systems remains a critical challenge. As a narrative survey rather than an original benchmark study, this paper reports no new experimental results; instead, it introduces a modality-aware analytical framework organizing the existing literature across four dimensions: imaging modality, data provenance, validation maturity, and model architecture. Using this taxonomy, the survey synthesizes unimodal and multimodal fusion approaches spanning radiology (CT, MRI, X-ray), pathology (whole-slide images), ophthalmology (fundus photography, OCT), and multi-source fusion combining imaging with electronic health records (EHR) and genomic data. The synthesis indicates that high reported accuracy is strongly contingent on data characteristics and evaluation conditions, with many models relying on low-maturity validation lacking evidence of generalization in real-world settings. To address these limitations, an engineering-oriented deployment framework is proposed, integrating modality-driven model selection, structured preprocessing pipelines, multi-level clinical validation, computational feasibility assessment, and explainability, together with a clinical deployment readiness model spanning validation maturity, data diversity, interpretability, and regulatory alignment. Key challenges include the single-site generalization gap, algorithmic bias across demographic groups, limited clinical adoption of explainable AI, insufficient alignment with regulatory frameworks including FDA 510(k), De Novo, and EU MDR/IVDR pathways, and a continuing need for prospective multicenter validation. Future directions toward federated learning, foundation models, certification-aware design, and multimodal digital biomarker integration are outlined.
Enoch Jacob Dodo, Amos Takai Yayock, Gregory Onwodi et al.· Journal of Science Research...· 0 citations
The Medical Segment Anything Model (MedSAM) has emerged as a prominent foundation model for medical image segmentation, prompting rapid growth in related research. Despite this proliferation, the literature remains fragmented across multimodal paradigms and lacks rigorous synthesis of methodological innovations, clinical safety evidence, algorithmic bias, and real-world implementation barriers—gaps that limit the field’s ability to translate algorithmic progress into clinical impact.
To systematically identify, classify, and synthesize published research on MedSAM and its derivatives, providing a comprehensive overview of methodological developments, evaluation findings, and clinical translation progress.
This scoping review followed the PRISMA-ScR guideline. We conducted a systematic search across six major databases (PubMed, Embase, Web of Science, Scopus, IEEE Xplore, ACM Digital Library) through June 2025.
From 1,482 initial records, 45 studies met inclusion criteria. Innovation-oriented studies (
n
= 32) were classified along four methodological axes spanning multimodal paradigms: Architecture Enhancement (encoder/decoder refinements for efficiency and precision), Prompt Strategy (textual, spatial, and multimodal prompting mechanisms), Efficiency and Deployability (quantization, distillation, and system integration), and Pipeline and Application (integration into clinical workflows). Evaluation-oriented studies (
n
= 13) revealed that MedSAM outperforms general-purpose SAM by 5–20% on structured modalities (CT/MRI) but lags behind task-specific networks for delicate structures. Notably, fairness analyses from two studies (combined
n
= 1,757 subjects) report Dice disparities of up to 10% across sex, age, and BMI strata, raising concerns about potential demographic bias that warrants further investigation in larger, multi-center cohorts before firm clinical-safety conclusions can be drawn. Real-world deployment is further constrained by prompt sensitivity (5–15% Dice variation), cross-modality domain shift, regulatory compliance gaps (FDA/CE marking, HIPAA, GDPR), and absent multi-center prospective validation, all of which represent principal implementation hurdles for clinical adoption.
MedSAM represents a significant advancement in medical image segmentation, demonstrating a clear trajectory from large-scale architectures toward lightweight, prompt-aware, and deployable models. This synthesis provides a structured critical appraisal indicating that the evidence base remains at the proof-of-concept stage, with most studies retrospective and single-center, and emerging signals of algorithmic bias, unresolved cross-modality failure modes, and incomplete regulatory pathways represent safety and implementation considerations that should be addressed before multimodal MedSAM-based systems are more broadly deployed in clinical imaging workflows.
Wenjie Zhang, Liming Luo, Mengnan He et al.· BMC Medical Imaging· 0 citations
BACKGROUND
Current artificial intelligence (AI) models for medical imaging predominantly focus on a single imaging modality and a single disease. Attempts to create multimodal and multi-disease models have resulted in inconsistent clinical accuracy. Furthermore, training these models typically requires large, well labelled datasets, which are costly and labour intensive to prepare. We aimed to train and evaluate an AI model that can interpret diverse imaging modalities across specialties while maintaining robust performance within each modality.
METHODS
We developed Multimodal, Multi-Disease Medical Imaging Foundation Model (MerMED-FM), a multi-specialty model trained using self-supervised learning and a memory module. MerMED-FM was pretrained on publicly sourced, unlabelled medical images from 12 specialties and seven imaging modalities: chest x-rays, CT, ultrasound, histopathology, colour fundus photography (CFP), optical coherence tomography (OCT), and dermatoscopy. After pretraining, the model was fine-tuned, validated, and evaluated for the diagnosis of a range of diseases on 26 public datasets and five private datasets comprising radiology, histopathology, and ophthalmology images. MerMED-FM was compared against a general-domain vision foundation model, various specialist single-modality foundation models, and a multispecialty foundation model. Models were fine-tuned using 10%, 30%, 50%, and 100% of data, with primary comparative analyses conducted using a 10% label fraction. The primary outcome was the area under the receiver operating characteristic curve (AUROC), which was summarised by imaging modality.
FINDINGS
MerMED-FM was trained on around 3·3 million images from 53 publicly available, unlabelled datasets, comprising 713 931 chest x-rays, 292 353 CT slices, 389 885 ultrasound frames, 1 017 712 pathology patches, 333 099 CFP images, 176 719 OCT slices, and 401 059 dermatoscopy images. Strong performance was achieved across all modalities at a label fraction of only 10%, with mean AUROC values of 0·844 for chest x-rays, 0·906 for CT, 0·818 for ultrasound, 0·908 for histopathology, 0·810 for CFP, 0·962 for OCT, and 0·827 for dermatoscopy.
INTERPRETATION
MerMED-FM has the potential to be a highly adaptable, versatile, cross-specialty foundation model that enables robust interpretation of medical imaging across diverse medical disciplines.
FUNDING
National Medical Research Council, Singapore and the Agency for Science, Technology and Research, Singapore.
Yang Zhou, C. Quek, Jun Zhou et al.· The Lancet Digital Health· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
