A Two-Stage Mutual Fusion Network for Multispectral and Panchromatic Image Classification

With the rapid development of remote sensing technology, satellites can easily obtain multispectral (MS) and panchromatic (PAN) images. How to mine the essence and peculiarity of the MS and PAN images and utilize their complementary to improve classification performance is still a challenge. This paper designs a two-stage mutual fusion network (TSMF-Net) for MS and PAN image classification. The network can be divided into two stages: data fusion and feature fusion. In the data fusion stage, we propose an adaptive twin intensity-hue-saturation (ATIHS) strategy. It not only aligns the size and channels of the MS and PAN images by a novel q-Split operation, but also introduces an adaptive soft-average mask to reduce the differences between replacement components, effectively mitigating spectral distortion and paving the way for the next stage. In the feature fusion stage, we propose a feature graft block (FG-Block) in which we introduce triplet loss and design an interlaced channel addition (ICA) module. Under the supervision of triplet loss, the FG-Block separates and hauls each branch’s essential and peculiar features. With the help of the ICA module, it can effectively graft the essential feature between branches and retain the peculiar feature of each branch, improving the utilization and discrimination of features. Finally, composed of the ATIHS, FG-Blocks, and output layers, our TSMF-Net is proven to improve the accuracy of the remote sensing classification task. The experimental results on multiple datasets verify the effectiveness of our proposed algorithms. Our code is available at: https://github.com/liaoyinuo/TSMF-Net.