TR-GAN: thermal to RGB face synthesis with generative adversarial network for cross-modal face recognition

Unlike RBG cameras, thermal cameras perform well under very low lighting conditions and can capture information beyond the human visible spectrum. This provides many advantages for security and surveillance applications. However, performing face recognition tasks in the thermal domain is very challenging given the limited visual information embedded in thermal images and the inherent similarities among facial heat maps. Attempting to perform recognition across modalities, such as recognizing a face captured in the thermal domain given the corresponding visible light domain ground truth database or vice versa is also a challenge. In this paper, a Thermal to RGB Generative Adversarial Network (TRGAN) to automatically synthesize face images captured in the thermal domain, to their RBG counterparts, with a goal of reducing current inter-domain gaps and significantly improving cross-modal facial recognition capabilities is proposed. Experimental results on the TUFTS Face Dataset using the VGG-Face recognition model without retraining, demonstrates that performing image translation with the proposed TR-GAN model almost doubles the cross-modal recognition accuracy and also performs better than other state-of-the-art GAN models on the same task. The generator in our network uses a UNET like architecture with cascaded-in-cascaded blocks to reuse features from earlier convolutions, which helps generate high quality images. To further guide the generator to synthesize images with fine details, we optimize a training loss as the weighted sum of the perceptual, adversarial, and cycle-consistent loss. Simulation results demonstrate that the proposed model generates more realistic and more visually appealing images, with finer details and better reconstruction of intricate details such sunglasses and facial emotions, than similar GAN models.