PFGAN: Fast transformers for image synthesis

Recently, the Transformers have shown great potential in computer vision tasks, such as classification detection, segmentation, and image synthesis, etc. The success of Transformers has been long attributed to the attention-based token mixer. However, the computational complexity of the attention-based token mixer module is quadratic to the number of tokens to be mixed. Therefore, the attention-based token mixer module requires more parameters and will cause a very large amount of computation. As far as image synthesis task is concerned, the attention-based token mixer module increases the computation amount of generative adversarial networks (GANs) based on Transformers. To address this problem, we propose the PFGAN method. The motivation is based on our observation that the computational complexity of pooling is linear to the sequence length, without any other learnable parameters. Based on this observation, we use pooling rather than self-attention as the token mixer. Experimental results on CelebA, CIFAR-10 and LSUN datasets demonstrate that our proposed method has fewer parameters and fewer computational complexity.