Exploring and Exploiting Multi-Modality Uncertainty for Tumor Segmentation on PET/CT

Despite the success of deep learning methods in multi-modality segmentation tasks, they typically produce a deterministic output, neglecting the underlying uncertainty. The absence of uncertainty could lead to over-confident predictions with catastrophic consequences, particularly in safety-critical clinical applications. Recently, uncertainty estimation has attracted increasing attention, offering a measure of confidence associated with machine decisions. Nonetheless, existing uncertainty estimation approaches primarily focus on single-modality networks, leaving the uncertainty of multi-modality networks a largely under-explored domain. In this study, we present the first exploration of multi-modality uncertainties in the context of tumor segmentation on PET/CT. Concretely, we assessed four well-established uncertainty estimation approaches across various dimensions, including segmentation performance, uncertainty quality, comparison to single-modality uncertainties, and correlation to the contradictory information between modalities. Through qualitative and quantitative analyses, we gained valuable insights into what benefits multi-modality uncertainties derive, what information multi-modality uncertainties capture, and how multi-modality uncertainties correlate to information from single modalities. Drawing from these insights, we introduced a novel uncertainty-driven loss, which incentivized the network to effectively utilize the complementary information between modalities. The proposed approach outperformed the backbone network by 4.53 and 2.92 Dices in percentages on two PET/CT datasets while achieving lower uncertainties. This study not only advanced the comprehension of multi-modality uncertainties but also revealed the potential benefit of incorporating them into the segmentation network. The code is available at https://github.com/HUST-Tan/MMUE .