Data and knowledge-driven deep multiview fusion network based on diffusion model for hyperspectral image classification

It is a crucial means for humans to perceive geomorphic features and landscape architectures by classifying ground objects in hyperspectral images (HSIs). Currently, the exponential development of neural networks has provided a powerful support for the accurate HSI classification. However, existing neural network-based methods usually rely solely on the data to drive the classification model, lacking attention to valuable land-cover distribution knowledge in HSIs. In view of this, to utilize hyperspectral data and distribution knowledge simultaneously, a data and knowledge-driven deep multiview fusion network based on diffusion model (DKDMN) is proposed in this paper. DKDMN extracts knowledge from unlabeled data in HSIs through a diffusion model-based knowledge learning framework (DMKLF), and combines raw hyperspectral data with the acquired knowledge through a designed deep multiview network architecture (DMNA) to mine complicated land-cover distribution information and reflect sample relationships. First, the proposed DMKLF utilizes the data distribution reconstructed by the diffusion model as a knowledge source for one view to enhance the network cross-sample awareness ability. On the other hand, the original HSI patches are considered a data source for another view, which co-drives DMNA with the unsupervised diffusion knowledge extracted by DMKLF to perform effective feature extraction. Second, taking into account the characteristics of each view and the feature similarity between these two views, a joint loss function specifically for DMNA is suggested to minimize the difference between the model predictions and the real labels. Finally, a multi-backbone integration classification framework (MBICF) is designed by deeply fusing three vision architectures to capture multi-scale spectral features and local–global features, thereby achieving pixel-wise classification effectively. Experimental results on four publicly available HSI datasets demonstrate that the proposed DKDMN achieves competitive classification accuracy compared with other state-of-the-art methods. For instance, the proposed DKDMN achieves an overall accuracy improvement of 1.62% and 2.18% on the Indian Pines and Salinas Valley datasets, respectively, compared to the multiple vision architecture-based hybrid network (MVAHN). The related code will be released at https://github.com/ZJier/DKDMN.