Multi-Scale Hyperspectral Pansharpening Network Based on Dual Pyramid and Transformer

Hyperspectral pansharpening is to fuse a high spatial resolution panchromatic image (PAN) with a low spatial resolution hyperspectral image (LR-HSI) and generate high resolution hyperspectral image (HR-HSI). However, most existing deep learning-based pansharpening methods have some issues, such as spectral distortion and insufficient spatial texture enhancement. In this work, we propose a novel multi-scale pansharpening network based on the Dual Gaussian- Laplacian Pyramid(DGLP) and Transformer, named MDTP-Net. Specifically, the DGLP module is designed to obtain feature maps at multi-level scales, which effectively learn global spectral information and spatial detail texture information. Then, we design a corresponding Transformer module for each scale feature and utilize the multi-head attention mechanism to guide the extraction of spatial information from LR-HSI and PAN images. This enhances the stability of pansharpening and improves the fusion of spectral with spatial information across feature spaces. In addition, the feature extractors are inserted to connect DGLP and Transformer, making the spatial feature map smoother and richer in channel and texture features. The feature fusion and multi-scale feature connection (MFC) blocks are used to connect multi-scale information together to generate HR-HSI images with more comprehensive spatial and spectral features. Finally, extensive experiments on three classic hyperspectral datasets are conducted. The experimental results demonstrate that our proposed MDTP-Net outperforms conventional methods and existing deep learning-based methods.