Polar coordinate sampling-based segmentation of overlapping cervical cells using attention U-Net and random walk

Segmentation of nuclei and cytoplasm inside the cellular clumps in cervical smear images is a difficult task because of the poor contrast and unpredictable shape of cytoplasm. This article addresses a new framework based on Attention U-Net (ATT U-Net) network and graph-based Random Walk (RW) to extract both nucleus and cytoplasm of each individual cell within an image of overlapping cervical cells. The proposed approach starts by separating nuclei from the cellular clumps through ATT U-Net architecture. Then, we remove fake nuclei that are usually much smaller than real nuclei. For each nucleus, a polar coordinate sampling matrix is generated. Each element in this matrix is realized by converting the image pixel from Cartesian coordinates to polar coordinates. After that, converted images would serve as the input of ATT U-Net for predicting cytoplasm. And finally, Graph-based RW is applied to extract the contour of cytoplasm. Because the features of cytoplasm boundaries in predicted maps are so obvious that the segmentation of every individual cell, including overlapping area, worked well under RW. We evaluate our framework on ISBI 2014 Challenge Dataset. The results reveal that our approach improves the performance on extracting individual cell from heavy overlapping cell clumps.