D2TNet: A ConvLSTM Network with Dual-direction Transfer for Pan-sharpening

In this paper, we propose an efficient convolutional long short-term memory (ConvLSTM) network with dual-direction transfer for pan-sharpening, termed as D2TNet. We design a specially structured ConvLSTM network that allows for dual-directional communication including multi-scale information and multi-level information. On the one hand, due to the sensitivity of spatial information to scales and the sensitivity of spectral information to levels, multi-scale and multi-level information are extracted to facilitate fuller use of source images. On the other hand, ConvLSTM is employed to capture the strong dependencies between multi-scale information and multi-level information. Besides, we introduce a multi-scale loss to enable different scales contributing to each other to generate high-resolution multi-spectral images that are closer to the ground truth. Extensive experiments, including qualitative evaluation, quantitative evaluation and efficiency comparison, are implemented to verify that our D2TNet outperforms state-of-the-art methods indeed.