Multi-Modal Medical Image Fusion Using Hybrid CNN-Transformer Models for Early Detection of Chronic Diseases

Abstract

Chronic disease early and accurate detection is a major healthcare issue nowadays, and most importantly, there is the rising prevalence or use of heterogeneous medical image data such as CT, MRI, X-ray, and retinal scans. Conventional models of deep learning such as CNNs perform well on spatial aspects of feature extraction but not generally on long-term relations and overall context. In this paper, we introduce a new hybrid deep learning network combining Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) to carry out multi-modal medical image fusion to assist in the diagnostic process with better results. The CNN branch gives fine-grain local representation and the Transformer module will predict the global connections between modalities. Empirical tests on publicly available data show that the proposed model is better than single CNN and Transformer models in using the datasets and there are massive increments in precise score, recall, and F1-score on chronic disease diagnosis in the initial stages. The field of research shows how hybrid architecture can be used to combine complementary information about various scanning modalities, which can become the direction of AI-aided decision-making in prevention.