Hyperspectral Pansharpening Based on Improved Deep Image Prior and Residual Reconstruction

Hyperspectral pansharpening aims to synthesize a low-resolution hyperspectral image (LR-HSI) with a registered panchromatic (PAN) image to generate an enhanced HSI with high spectral and spatial resolution. Recently, the proposed HS pansharpening methods have obtained remarkable results using deep convolutional networks (ConvNets), which typically consist of three steps: 1) upsampling the LR-HSI; 2) predicting the residual image via a ConvNet; and 3) obtaining the final fused HSI by adding the outputs from first and second steps. Recent methods have leveraged deep image prior (DIP) to upsample the LR-HSI due to its excellent ability to preserve both spatial and spectral information, without learning from large datasets. However, we observed that the quality of upsampled HSIs can be further improved by introducing an additional spatial-domain constraint to the conventional spectral-domain energy function. We define our spatial-domain constraint as the $L_{1}$ distance between the predicted PAN image and the actual PAN image. To estimate the PAN image of the upsampled HSI, we also propose a learnable spectral response function (SRF). Moreover, we noticed that the residual image between the upsampled HSI and the reference HSI mainly consists of edge information and very fine structures. In order to accurately estimate fine information, we propose a novel overcomplete network, called HyperKite, which focuses on learning high-level features by constraining the receptive from increasing in the deep layers. We perform experiments on three semisynthetic and one real HSI datasets to demonstrate the superiority of our DIP-HyperKite over the state-of-the-art pansharpening methods. The deployment codes, pretrained models, and final fusion outputs of our DIP-HyperKite and the methods used for the comparisons will be publicly made available at https://github.com/wgcban/DIP-HyperKite.git .