Modality Conversion Meets Super-Resolution: A Collaborative Framework for High-Resolution Thermal UAV Image Generation

Due to the limitations and costs of thermal sensors, unmanned aerial vehicle (UAV) platforms often equip with high-resolution (HR) visible imaging and low-resolution (LR) thermal imaging cameras for all-day monitoring capability. Existing works generate the high-resolution thermal UAV images by either super-resolution (SR) from high-resolution visible and low-resolution thermal images or modality conversion (MC) from high-resolution visible images. However, the modality gap between visible and thermal sources might degrade the generation quality. We observe that the MC task is beneficial in addressing the cross-modal gap in the SR task, while the SR task can provide the condition of thermal information to boost the MC task. Moreover, these two tasks have the same output and can thus be carried out simultaneously without any additional annotation. Based on this observation, we propose a collaborative enhancement network (CENet), which performs thermal UAV image SR and visible image MC in a joint manner, for high-resolution thermal UAV image generation. In particular, we design a mutual guidance module to interact the features from SR and MC tasks in an alternating bidirectional manner. Considering that low-level vision tasks are position-sensitive, to further enhance the feature alignment between the two tasks, we design a bidirectional alignment fusion module to maintain feature consistency of the MC and SR branches. The proposed collaborative framework not only achieves joint and unified training of the two tasks, but also generates two types of complementary high-resolution images. Extensive experiments on public datasets demonstrate that the proposed CENet outperforms current state-of-the-art super-resolution (SR) methods in generating high-resolution thermal UAV images, as quantified by peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM).