Multi-view Image Generation by Cycle CVAE-GAN Networks

In this paper, we address the problem of multi-view image generation from a single view. To this end, we investigate Variational Auto-Encoder (VAE) and Generative Adversarial Network (GAN) as effective solutions to this problem. Inspired by CycleGAN playing an important role in image-to-image translation applications, we utilize the idea of cycle consistency to generate images in multi-view. With VAE and GAN as the basic components, we propose Cycle Conditional-VAE-GAN (Cycle CVAE-GAN) to tackle the problem within four steps. First, the source image with target-view condition and the target image are both mapped to the shared latent variable space by two encoders. Second, we sample a variable from the shared variable space as the input of a designed decoder for the low-resolution target image generation. Third, we repeat the previous two steps. It is worth mentioning that the inputs of those two encoders are the generated low-resolution target image with source-view condition and the source image. Then the reconstructed source image can contribute to the cycle consistency loss. Finally, a GAN framework with a dual-input U-Net generator and a patch discriminator are proposed to generate high-resolution and realistic target images. Experiments on the Multi-View Clothing (MVC) dataset demonstrate that the proposed method achieves better results than the state-of-the-art models.