Automated Nucleus Segmentation in Histology Images: An Attention U-Net Approach with Comprehensive Preprocessing and Inference Pipelines

Abstract

Objectives: In the area of digital pathology, nucleus segmentation is an essential step for quantitative analysis; however, high nuclear density and staining heterogeneity in histology images pose a great challenge. Methods: This study implements a lightweight Attention U-Net model on the MoNuSeg 2018 dataset to overcome these challenges. The proposed pipeline integrates comprehensive image and mask preprocessing, including stain deconvolution and multi-pass mask smoothing, with a high-fidelity overlapping patch-stitching strategy during inference on high-resolution images. Results: The model achieved an Intersection over Union (IoU) score of 0.8745 on full-resolution, unseen test images. This result demonstrates the effectiveness of attention mechanisms in enhancing feature localization and validates the stitching methodology for generating seamless and accurate segmentation maps. Conclusion: Attention U-Net is an effective and competitive method for automated, scalable nucleus delineation in challenging clinical contexts.