Deep learning-based multi-modal imaging for early detection of brain tumour

Abstract

Early and accurate discovery of brain tumours is crucial for timely diagnosis and effective treatment planning. In this work, we put forward an explainable multimodal CNN–Transformer framework, kind of merges MRI and CT cues in a complementary way, using attention-guided multimodal fusion so that tumour classification and localisation can happen at the same time. Instead of most existing multimodal CNN–Transformer methods, which mostly linger on feature merging for classification only, this proposal does a bit more. It uses modality-specific CNN encoders, a Transformer based cross-modal attention module, and several fusion recipes—early fusion, late fusion, and attention-driven fusion. On top of that we do quantitative Grad-CAM checking against expert radiologist annotations, plus a computational efficiency examination, all inside a single coherent architecture. Overall this should boost interpretability and make the approach more clinically usable. For training and evaluation, the framework was tested with a carefully curated dataset of 1,986 patient-level MRI–CT pairs collected from BraTS 2021 and the TCIA Brain Tumour CT Archive. The dataset includes glioma, meningioma, pituitary tumour, and healthy controls. We used a patient-wise split: 70% for training, 15% for validation, and 15% for testing. When we looked at the attention based fusion approach, it delivered classification accuracy of 96.2 ± 0.3% and an AUC of 0.98. For localisation, Grad-CAM alignment produced a Dice overlap of 0.84 ± 0.04 versus annotations from three radiologists, across five separate experimental runs (reported as mean ± SD). Beyond the metrics, the framework also looks practical for real-world deployment. Inference time was 0.18 s per image, computational complexity reached 9.3 GFLOPs, and GPU memory use was 5.6 GB. So, these findings suggest that the proposed framework delivers an accurate, understandable, and compute friendly solution for multimodal brain tumour diagnosis, with clear benefits for clinical decision support.