CIMFNet: Cross-Layer Interaction and Multiscale Fusion Network for Semantic Segmentation of High-Resolution Remote Sensing Images

Semantic segmentation of remote sensing images has received increasing attention in recent years; however, using a single imaging modality limits the segmentation performance. Thus, digital surface models have been integrated into semantic segmentation to improve performance. Nevertheless, existing methods based on neural networks simply combine data from the two modalities, mostly neglecting the similarities and differences between multimodal features. Consequently, the complementarity between multimodal features cannot be exploited, and excess noise is introduced during feature processing. To solve these problems, we propose a multimodal fusion module to explore the similarities and differences between features from the two information modalities for adequate fusion. In addition, although downsampling operations such as pooling and striding can improve the feature representativeness, they discard spatial details and often lead to segmentation errors. Thus, we introduce hierarchical feature interactions to mitigate the adverse effects of downsampling and introduce a two-way interactive pyramid pooling module to extract multiscale context features for guiding feature fusion. Extensive experiments performed on two benchmark datasets show that the proposed network integrating our novel modules substantially outperforms state-of-the-art semantic segmentation methods. The code and results can be found at https://github.com/NIT-JJH/CIMFNet .