MMGLOTS: Multi-Modal Global-Local Transformer Segmentor for Remote Sensing Image Segmentation

Multi-modal semantic segmentation of remote sensing (RS) images is a challenging task due to the complex relationship between different modalities and the large intra-class variance of objects in RS images. Existing semantic segmentation methods can only utilize the information of a single modality, which is not sufficient to obtain accurate segmentation results. To address this problem, in this paper, a novel multimodal global-local transformer segmentor (MMGLOTS) is proposed to cope with the multi-modal semantic segmentation task. Specifically, the semantic features of each modality are extracted by the multi-modal semantic feature extractor (MMSFE) with an adaptive fusion strategy. Then, the features are aggregated, and deep representations of both local and global dependencies are obtained by the global-local transformer (GLT). The final prediction is obtained by progressively restoring the deep representations with a prediction restorer (PR). Extensive experiments on two multi-modal semantic segmentation datasets show that our method achieves superior performance and the proposed method achieves the first place on the newly held Cross-City Multi-modal Semantic Segmentation Challenge 2023.