Hybrid U-Net++–Vision Transformer Fusion for Accurate Brain MRI Image Segmentation and classifications

Abstract

Automatic brain Magnetic Resonance Imaging (MRI) analysis plays a crucial role in computer-aided diagnosis, treatment planning, and disease monitoring by enabling accurate delineation and identification of brain tumors. However, manual segmentation is labor-intensive, time-consuming, and susceptible to inter-observer variability, making automated and reliable methods highly desirable. This paper proposes a Hybrid U-Net++ and Vision Transformer (ViT) Fusion framework for automatic brain MRI tumor segmentation and segmentation-guided classification. The proposed architecture employs a U-Net++ encoder with nested skip connections to extract hierarchical multi-scale features, while a Vision Transformer captures long-range spatial dependencies and global contextual information through self-attention mechanisms. A multi-stage feature fusion strategy integrates convolutional and transformer representations, complemented by boundary-aware refinement to improve tumor localization and preserve fine anatomical details. The segmentation model achieved a training Dice score of 95.6%, IoU of 91.4%, and validation Dice score of 92.6%, demonstrating stable convergence and strong generalization capability. Furthermore, the proposed framework attained a Dice score of 95.8%, IoU of 92.1%, and Hausdorff Distance of 3.35 mm in the ablation analysis, confirming the effectiveness of multi-stage feature fusion and boundary-aware refinement. The segmented tumor regions are subsequently utilized for Region of Interest (ROI)-based classification using deep feature extraction, Global Average Pooling (GAP), and fully connected layers. The classification module distinguishes glioma, meningioma, pituitary tumor, and no-tumor cases, achieving an overall accuracy of 97.55%, macro-average precision of 97.13%, recall of 97.50%, F1-score of 97.31%, and specificity of 98.40%. Comparative analysis further demonstrates the effectiveness of the proposed hybrid architecture over conventional CNN-based and transformer-based approaches. The combined segmentation and classification results indicate that the proposed framework provides an effective approach for tumor localization and tumor-type identification, with potential applicability in automated brain MRI analysis and clinical decision support.