Brain Tumor Segmentation Using Diffusion Transformers

Abstract

Brain tumor segmentation from magnetic resonance imaging (MRI) is central to sur-gical planning and radiotherapy targeting, yet manual delineation is time-intensiveand subject to significant inter-rater variability. Dominant transformer-based seg-mentation architectures require 60–150 million parameters and exceed 300 GFLOPsper inference pass, while diffusion-based segmentation approaches demand 50–1,000network evaluations per prediction, placing both categories beyond the hardwarebudget of resource-constrained clinical settings.We propose an efficient diffusion-based segmentation framework combining thefollowing components. An Hourglass Diffusion Transformer (HDiT) backbone thatemploys neighborhood attention at high-resolution stages and global self-attentionat the bottleneck, achieving linear computational complexity in full pixel space. Adedicated Hierarchical Feature Encoder (HFE) extracts multi-scale MRI represen-tations and injects scale-aligned features at corresponding hierarchical levels of theHDiT backbone, providing anatomical guidance throughout the denoising process.The framework adapts the Elucidating Diffusion Models (EDM) preconditioningscheme to medical image segmentation for the first time, pairing it with a compoundloss combining EDM preconditioning with Dice supervision. The stochastic natureof diffusion inference additionally enables implicit test-time ensembling, which im-proves segmentation quality, and yields clinically relevant uncertainty estimates.On the BraTS 2020 test set, the method achieved 84.07% mean Dice SimilarityCoefficient (DSC) using 13.66 million parameters and 7.14 GFLOPs per inferencestep. Compared to competing architectures in the literature, this represents equiv-alent segmentation accuracy with substantially fewer parameters and significantlylower per-step computational cost, demonstrating that high-quality diffusion-basedsegmentation is achievable within practical resource constraints.These results demonstrate that the proposed framework achieves segmentationaccuracy competitive with larger architectures at substantially lower computationalcost. Beyond accuracy, stochastic ensembling yields uncertainty maps that flagambiguous boundaries for targeted clinical review, a property requiring no additionalarchitectural overhead. Together, these findings position compact diffusion-basedsegmentation as a scalable and practical framework for future clinical deployment.