A Dual-Branch Multiscale Transformer Network for Hyperspectral Image Classification

In recent years, convolutional neural networks (CNNs) have achieved great success in hyperspectral image (HSI) classification tasks. CNNs focus more on the local features of HSIs. The recently emerging Transformer network has shown great interest in the global features of HSIs. However, existing Transformer networks only consider single-scale feature extraction and do not combine the advantages of multiscale feature extraction and Transformer global feature extraction. To address this issue, this article proposes a dual-branch multiscale Transformer (DBMST) for HSI classification. First, a large-size spectral convolution kernel is utilized for the spectral dimension of the hyperspectral cube for downsampling feature extraction. Next, a channel shrink soft split module (CS3M) is proposed, which not only solves the problem of missing local information in large-scale tokens but also extracts shallow features and performs dimensionality reduction on channels. Then, considering the different dimensions of features extracted at different scales in two branches, a pooled activation fusion module (PAFM) is carefully designed. Finally, the proposed DBMST is evaluated on three commonly used HSI datasets. The experimental results show that DBMST achieves better classification performance compared to other advanced networks, demonstrating the effectiveness of the proposed method in HSI classification.