Accurate segmentation of overlapping cells in cervical cytology with deep convolutional neural networks

Accurate cell segmentation is essential for computer-aided diagnosis of cervical precancerous lesions in cytology images. Automated segmentation poses a great challenge due to the presence of fuzzy and overlapping cells, noisy background, and poor cytoplasmic contrast. Deep learning diagnosis technology has showed its advantages in segmenting complex medical images. We present a new framework based on deep convolutional neural networks (DCNNs) to automatically segment overlapping cells in digital cytology. A double-window based cellular detection method is derived to correctly localize individual cells, in which TernausNet is adopted to classify the image pixels into nucleus, cytoplasm, or background. A modified DeepLab V2 model is applied to perform cytoplasm segmentation. To provide more training samples, a synthesis method is utilized to generate cell masses containing touching or overlapping cells. The presented method was tested on three independent data cohorts, including two public datasets. We achieved improved performance in terms of dice coefficient (DSC), false negative and false positive rates, with up to 15% improvement in DSC, compared with the state-of-the-art approaches. The results indicated that the DCNN based segmentation method could be useful in an image-based computerized analysis system for early detection of cervical cancer.