Model-informed Multi-stage Unsupervised Network for Hyperspectral Image Super-resolution

By fusing a low-resolution hyperspectral image (LrMSI) with an auxiliary high-resolution multispectral image (HrMSI), hyperspectral image super-resolution (HISR) can generate a high-resolution hyperspectral image (HrHSI) economically. Despite the promising performance achieved by deep learning (DL), there are still two challenges remaining to be solved. First, most DL-based methods heavily rely on large-scale training triplets, which reduces them to limited generalization and poor practicability in real-world scenarios. Second, existing methods pursue higher performance by designing complex structures from off-the-shelf components while ignoring inherent information from the degradation model, hence leading to insufficient integration of domain knowledge and lower interpretability. To address those drawbacks, we propose a model-informed multi-stage unsupervised network, M2U-Net for short, by leveraging both deep image prior (DIP) and degradation model information. Generally, M2U-Net is built with a three-stage scheme, i.e., degradation information learning (DIL), initialized image establishment (IIE), and deep image generation (DIG) stages. The first stage is to exploit the deep information of the degradation model via a tiny network whose parameters and outputs will serve as guidance for the following two stages. Instead of feeding uninformed noise as input for stage three, IIE stage aims to establish an initialized input with expressive HrHSI-relevant information by resorting to a spectral mapping learning network, thus facilitating the extraction of prior information and further magnifying the potential of DIP for high-quality reconstruction. Last, we propose a dual U-shape network as a powerful regularizer to capture image statistics, in which two U-Nets are coupled together by cross-attention guidance (CAG) module to separately achieve spatial feature extraction and final image generation. The CAG module can incorporate abundant spatial information into the reconstruction process and hence guide the network toward a more plausible generation. Extensive experiments demonstrate the effectiveness of our proposed M2U-Net in terms of quantitative evaluation and visual quality. The code will be available at https://github.com/JiaxinLiCAS.