Nuclei segmentation using attention aware and adversarial networks

Accurate segmentation of nuclei plays a critical role in pathology since assessments and diagnoses are mainly based on the recognition, measurement, and counting of nuclei. However, in digital pathology, automated nucleus segmentation is a challenging issue because of various factors such as color inconsistency in stained images, unclear nucleus boundaries, low contrast between background and nuclei, different shapes and sizes of nuclei, and intensity inhomogeneity not only inside nuclei but also across them and the background. In this work, an efficient method has been developed for nuclei segmentation. Its efficiency has been achieved through tissue and chemical invariant normalization, feature extraction with dense convolution layers, merging of local and global features, mask generation, and adversarial learning. For fair comparisons, state-of-the-art nuclei segmentations have been employed to the same data sets, and their performances have been evaluated using the same metrics. This paper's main contributions are threefold: (i) Introducing a novel technique for nuclei segmentation using an adversarial network and a hybrid attention-aware network. (ii) Presenting the effective merging of global and local features to enhance pattern recognition, and the utilization of efficient hybrid attention blocks for extracting desired global information and improving relationships between feature regions at different locations. (iii) Presenting experimental results showing that the proposed technique accomplishes nuclei detection and extraction with higher accuracy (a minimum improvement of 3.7%) than other recent methods. Also, each stage provides considerable contributions to the segmentation performance. Particularly, the hybrid attention-aware network has improved the performance by 4.2% according to the dice coefficient.