Hybrid Generative Adversarial Networks for Deep MR to CT Synthesis Using Unpaired Data

Many different methods have been proposed for generation of synthetic CT from MR images. Most of these methods depend on pairwise aligned MR and CT training images of the same patient, which are difficult to obtain. 2D cycle-consistent Generative Adversarial Networks (2D-cGAN) have been explored before for generating synthetic CTs from MR images but the results are not satisfied due to spatial inconsistency. There exists attempt to develop 3D cycle GAN (3D-cGAN) for image translation but its training requires large number of data which may not be always available. In this paper, we introduce two novel mechanisms to address above mentioned problems. First, we introduce a hybrid GAN (hGAN) consisting of a 3D generator network and a 2D discriminator network for deep MR to CT synthesis using unpaired data. We use 3D fully convolutional networks to form the generator, which can better model the 3D spatial information and thus could solve the discontinuity problem across slices. Second, we take the results generated from the 2D-cGAN as weak labels, which will be used together with an adversarial training strategy to encourage the generator’s 3D output to look like a stack of real CT slices as much as possible. Experimental results demonstrated that our approach achieved better results than the state-of-the-art when limited number of unpaired data are available.