Multi-Stage Feature Fusion for Breast Cancer Histopathology using EfficientNetV2 and Vision Transformers

Abstract

Early detection of breast cancer is essential for improving patient survival and reducing mortality. Although histopathology-based diagnosis provides definitive clinical evidence, the complex and heterogeneous nature of breast cancer histopathology (BCH) images poses significant challenges for accurate classification. Conventional deep learning models often struggle to capture both local tissue characteristics and global contextual patterns simultaneously. To address this limitation, this study proposes a Multi-Stage Feature Fusion (MSFF) model that integrates EfficientNetV2, a CNN local texture branch, and a Vision Transformer (ViT). EfficientNetV2 extracts high-level semantic features, the CNN branch captures fine-grained local texture information, and the ViT models long-range contextual dependencies through self-attention. These complementary representations are fused through an adaptive multi-stage fusion strategy to enhance discriminative capability for binary BCH classification. An ablation study is conducted to systematically evaluate the contribution of each architectural component. Experimental results on the BreakHis dataset demonstrate that the proposed model achieves the best overall balanced classification performance, with an accuracy of 93.04%, precision of 93.08%, recall of 92.69%, F1-score of 92.89%, and an AUC of 0.9303 at the optimal 200× magnification level. The findings confirm that the proposed fusion strategy provides balanced classification performance, stable optimization behaviour, and strong generalization capability, effectively addressing BCH image heterogeneity and offering strong potential for clinical decision-support applications.