Siamese semi-disentanglement network for robust PET-CT segmentation

A robust PET-CT segmentation network should guarantee that models trained on the PET-CT images will still work when only CT images are available. It is particularly important due to the radioactivity and expensive cost of PET imaging, in many cases only CT images can be obtained. Disentanglement and Generative Adversarial Networks (GAN) are two commonly used strategies to deal with the missing modality. Disentanglement methods cannot successfully disentangle PET-CT images into modal features and anatomical features because PET-CT images do not satisfy anatomical information consistency constraints. GAN networks tend to ignore information that is critical for downstream tasks, such as tumor information. To address above issues, we propose a siamese semi-disentanglement network. We extract high-level shared tumor features from PET images and CT images instead of anatomical features for downstream segmentation tasks. Meanwhile, in order to leverage low-level entanglement features during segmentation, GAN is used to generate synthetic PET images from CT images. Siamese Consistency Module (SCM) is proposed to ensure that the entanglement low-level features of the synthetic PET images are consistent with the real PET images. The motivation of our proposed method is that the entanglement information discarded by the semi-disentanglement is compensated by GAN to get rid of the anatomical information consistency constraints. Also, the GAN can better retain tumor information through semi-disentanglement. We do experiments on two public PET-CT datasets and one private dataset: Soft-Tissue-Sarcoma (STS) dataset, HeadNeck dataset and LiverTumor dataset. The results show that our proposed method can successfully achieve robust PET-CT segmentation. Our proposed method outperforms other disentanglement methods and generative networks in the absence of PET modality. In the inference stage, with missing PET images, using the siamese semi-disentanglement network proposed in this paper can achieve comparable results to the full modal segmentation.