Reciprocal transformer for hyperspectral and multispectral image fusion

Hyperspectral and multispectral (HS-MS) image fusion aims to reconstruct high-resolution hyperspectral images from low-resolution hyperspectral images and high-resolution multispectral images. Despite the popularity of convolutional neural networks in HS-MS fusion tasks, their potential is curtailed due to the limited receptive field of each neuron, resulting in inadequate long-range modeling capabilities. Although several Transformer-based HS-MS fusion methods have been proposed, most of them have not fully integrated and coordinated the data from the two modalities (i.e., hyperspectral images and multispectral images). This results from either merging the hyperspectral images and multispectral images at the inception stage or at the feature level after independent feature extraction. Such ineffective interactions significantly compromise the quality of the reconstructed hyperspectral images. In this paper, we introduce a reciprocal fusion strategy, i.e., the dual cross Transformer-based fusion (DCTransformer), for HS-MS fusion. The model excels in manipulating the interplay between the data streams of various modalities. A pivotal component of our model is the directional pairwise multi-head cross-attention, which concentrates on the interactions between multimodal sequences and can potentially facilitate the transfer of information from one modality to another. Additionally, we incorporate a Swin Transformer block post cross-attention to enhance the self-attention within the context. Extensive experiments show that our DCTransformer performs favorably against other recent works on both simulation HSI datasets and real HSI datasets. The source code and pre-trained models will be released.