Nuclei segmentation in histopathology images using Swin-UNet with explicit edge supervision

Abstract

Accurate nuclei segmentation in histopathology images is crucial for downstream biological analysis. While Transformer based models excel at capturing global contexts, they struggle with blurred boundaries and structural adhesion in dense cellular regions. To address this, we propose Swin-UNet-EARM, introducing an Edge-Aware Refinement Module (EARM) for feature-level decoupling. EARM extracts local high-frequency features via depth-wise separable convolutions within a dual-branch structure: one branch uses dynamic morphological gradients for explicit topological supervision, while the other leverages this signal to adaptively weight deep semantic features using spatial attention. Evaluations on the public MoNuSeg dataset demonstrate that our model outperforms the baseline Swin-UNet, increasing the Dice coefficient from 0.782 to 0.808 and the Aggregated Jaccard Index (AJI) from 0.51 to 0.57. Qualitative and ablation analyses confirm that EARM effectively mitigates cell adhesion and achieves fine-grained boundary delineation, providing a lightweight, robust solution for high-precision automatic nuclei segmentation.