Sep 2026· Wasit Journal of Computer and Mathematics Science· Vol 5, pp. 12-25· 0 citations· 4 references
TL;DR
The proposed Multimodal Adaptive Fusion Transformer (CMAFT) is a unified architecture that combines Swin transformer encoders for each media with a Catalyst Cross Attention Fusion Module (GCAF) to dynamically weight each media's contribution based on its real-time reliability.
Abstract
The rise of artificial intelligence technologies and their increasing reliance in numerous fields, most notably medicine, is significant. AI plays a crucial role in assisting physicians with accurate patient diagnosis, enabling them to make informed decisions. This article presents a method for integrating computer vision technologies into the medical field, where multimodal medical imaging integrates structural and textual information to support clinical decision-making. However, most image fusion frameworks based on transformers remain narrowly designed for a single media combination and a single subsequent task. This paper proposes the Multimodal Adaptive Fusion Transformer (CMAFT), a unified architecture that combines Swin transformer encoders for each media with a Catalyst Cross Attention Fusion Module (GCAF) to dynamically weight each media's contribution based on its real-time reliability. CMAFT was evaluated against two structurally different multimodal reference standards: the BraTS 2021 dataset, in which four MRI sequences (T1, T1ce, T2, FLAIR) were combined to segment brain tumor volumes, and the MIMIC-CXR dataset, in which chest X-ray images were combined with duplex radiology report inclusions to classify multimodal pathology and generate reports. The proposed GCAF module provides a learnable reliability gateway that curbs degraded or lost data streams, while a multiscale hierarchical integration pyramid preserves both fine detail and overall semantic context. Experiments show that CMAFT achieves average Dice scores of 92.4%, 87.9%, and 84.6% for whole tumor, tumor nucleus, and enhanced tumor, respectively, on the BraTS 2021 dataset, and a multi-class average area under the curve (AUC) of 0.876 with an overall F1 value of 0.658 on the MIMIC-CXR dataset, outperforming modern hybrid CNN-Transformer models while using fewer parameters compared to similar dual-code baseline models. Resection studies confirm that both reliability gate and multi-scale integration contribute to measurable gains, and that CMAFT degrades smoothly when either an MRI sequence or radiology report is unavailable. These results suggest that a single adaptive mutual attention integration design can be generalized across volumetric and text-based multimodal diagnostic pathways.
Introduction Medical image classification is fundamental to computer-aided diagnosis. Limited labeled samples, subtle inter-class differences, and heterogeneous lesion morphology make it difficult for a single representation to capture all relevant cues. This study investigates MedFuse, a simple dual-stream framework t...
Ya-Jing Ren, Hai Ling, Zheng Gu et al.· Frontiers in Medicine· 0 citations
The combination of digital medical imaging technology and computer vision is extremely important for the correct diagnosis of the human body. Different imaging technologies such as Computed Tomography (CT), Single Photon Emission Computed Tomography (SPECT), Positron Emission Tomography (PET), Magnetic Resonance Imagin...
Ranadhir Das, Arunima Bhattacharyya, Ashutosh Kumar Jha et al.· 2026 International Conferenc...· 0 citations
This survey organizes transformer-based fusion models for healthcare along four axes:modality, pairing, fusion level, and attention and architecture design, to give researchers and clinicians a structured entry point into this area.
N. Jagtap, D. J. Saini, Prasun Chakrabarti· Journal of Data Science and...· 0 citations
A lightweight Vision Transformer UNet is proposed that combines the hierarchical feature extraction capability of UNet with the global context modeling of Vision Transformers, enabling effective learning of both local and global features while maintaining computational efficiency with only 2.6 million trainable paramet...
Sheekar Banerjee, Monika Chowdhury, M. Akash et al.· arXiv.org· 0 citations
Brain tumors remain among the most life-threatening neurological disorders, and their outcome depends heavily on how early and how accurately they are diagnosed. Magnetic Resonance Imaging (MRI) is the modality of choice for tumor evaluation, yet manual reading of MRI scans is slow and subject to inter-observer disagre...
Sachin U. Kadam, Brijendra Gupta· Iconic research and engineer...· 0 citations