Joint Classification of Hyperspectral and LiDAR Data Using Height Information Guided Hierarchical Fusion-and-Separation Network

Hyperspectral image (HSI) and LiDAR data are complementary to each other, which can be combined to improve the classification performance. However, existing deep network models do not sufficiently consider their complementarity to design the network structure and loss functions. Moreover, there lacks a hierarchical mutual-assistance learning mechanism that leverages the modality-shared features to enhance the modality-specific ones and vice versa. In view of these, we propose a novel height information guided hierarchical fusion-and-separation network (HFSNet) for joint classification of HSI and LiDAR data. HFSNet consists of three major components, i.e., dual-structure feature encoders (DSFEs), feature fusion-and-separation blocks (F2SBs), and an edge decoder (ED). Specifically, the transformer and convolutional neural network are introduced in DSFEs to encode the spectral and spatial information of HSI and LiDAR data, respectively. In F2SBs, the deformable convolution-based height information guided fusion module and the modality separation refinement module are proposed to sequentially extract modality-shared and modality-specific features. Additionally, the ED is incorporated into our model to predict the LiDAR edge map from the HSI feature to improve the model's generalization ability. As such, the learned features from HSI and LiDAR data are deeply fused and mutually enhanced. Experiments on three benchmark datasets show the superiority of HFSNet to the state-of-the-art methods for jointly classifying HSI and LiDAR data with limited training samples.