Reciprocal transformer for hyperspectral and multispectral image fusion

Hyperspectral and multispectral (HS–MS) image fusion aims to reconstruct high-resolution hyperspectral images from low-resolution hyperspectral images and high-resolution multispectral images. While convolutional neural networks have been widely used for HS–MS fusion, their potential is curtailed due to the limited receptive field of each neuron, resulting in inadequate long-range modeling capabilities. Although several Transformer-based HS–MS fusion methods have been proposed, most of them often fail to fully integrate and coordinate the data from the two modalities (i.e., hyperspectral images and multispectral images). Such ineffective interactions significantly compromise the quality of the reconstructed hyperspectral images. In this paper, we introduce a novel reciprocal fusion strategy called the dual cross Transformer-based fusion (DCTransformer) for HS–MS fusion. The model excels in capturing the interplay between different modalities by utilizing directional pairwise multi-head cross-attention, which facilitates the transfer of information between modalities. Additionally, we incorporate a Swin Transformer block post cross-attention to enhance the self-attention within the context. Extensive experiments show that our DCTransformer performs favorably against other recent works on both simulation HSI datasets and real HSI datasets. The source code and pre-trained models are availabe at https://github.com/qingma2016/DCTransformer.