DBCTNet: Double Branch Convolution-Transformer Network for Hyperspectral Image Classification

Currently, deep learning methods represented by convolutional neural networks (CNNs) or Transformers are of great interest in hyperspectral image (HSI) classification. And recent works show that hybrid models using CNN and Transformer modules are expected to achieve better performance than when they are used alone. However, these hybrid models applied to HSI classification consider the combination of 2D CNN and Transformer, which makes the models have high computational complexity. And the information of multiple spectral dimensions different from ordinary RGB images has not been fully excavated. Based on this, we propose DBCTNet, a double branch Convolution-Transformer network. Specifically, a MSpeFE module is used for multiscale spectral feature extraction at the early stage of the proposed network. Then a ConvTE block is designed to improve the original Transformer encoder, where a Conv spectral projection unit and a convolutional multihead self-attention (CMHSA) unit are proposed to extract spatial and global spectral features. A double branch module is further built based on 3D CNN and ConvTE. This module can fully integrate spatial and local-global spectral features, while also having low computational complexity. Experiment results on four public datasets, Pavia University, Houston, WHU-Hi-LongKou and HuangHeKou, show that DBCTNet achieves satisfactory performance with a small number of parameters and relatively excellent efficiency compared to nine other networks. The implement of DBCTNet will be available publicly at https://github.com/xurui-joei/DBCTNet.