Unified medical image segmentation by learning from uncertainty in an end-to-end manner

Automatic segmentation is a fundamental task in computer-assisted medical image analysis. Convolutional neural networks (CNNs) have been widely used for medical image segmentation tasks. Currently, most deep learning-based methods output a probability map and use a hand-crafted threshold to generate the final segmentation result, while how confident the network is of the probability map remains unclear. The segmentation result can be quite unreliable even though the probability is much higher than the threshold since the uncertainty of the probability can also be high. Moreover, boundary information loss caused by consecutive pooling layers and convolution strides makes the object’s boundary in segmentation even more unreliable. In this paper, we propose an uncertainty guided network referred to as UG-Net for automatic medical image segmentation. Different from previous methods, our UG-Net can learn from and contend with uncertainty by itself in an end-to-end manner. Specifically, UG-Net consists of three parts: a coarse segmentation module (CSM) to obtain the coarse segmentation and the uncertainty map, an uncertainty guided module (UGM) to leverage the obtained uncertainty map in an end-to-end manner, and a feature refinement module (FRM) embedded with several dual attention (DAT) blocks to generate the final segmentations. In addition, to formulate a unified segmentation network and extract richer context information, a multi-scale feature extractor (MFE) is inserted between the encoder and decoder of the CSM. Experimental results show that the proposed UG-Net outperforms the state-of-the-art methods on nasopharyngeal carcinoma (NPC) segmentation, lung segmentation, optic disc segmentation and retinal vessel detection.