Local Climate Zone Classification via Semi-Supervised Multimodal Multiscale Transformer

Local climate zone (LCZ) classification plays a critical role in urban environment research and has attracted extensive attention from many researchers. However, the potential of deep learning-based approaches is not yet fully explored in this field, even though neural networks continue to push the frontier for various applications. In this paper, we propose a novel multimodal multiscale Transformer network for LCZ classification by introducing multiscale patch embedding and multimodal fusion learning in Transformer architecture. The proposed multiscale patch embedding effectively captures hierarchical interrelationships of image contextual neighborhoods, and automatically learns discriminative features. And the proposed multimodal fusion learning enables the network to naturally fuse multispectral and synthetic aperture radar (SAR) data under the guidance of attention mechanism. To further improve classification accuracy, we impose semi-supervised learning to mine unlabeled image data information. Both labeled and pseudo-labeled data jointly drive our network updates. Experiments conducted on the So2Sat LCZ42, CHN15-LCZ and SouthKorea6-LCZ benchmark datasets demonstrate that our proposed approach outperforms other existing methods significantly and achieves state-of-the-art performance. In the generated LCZ maps, urban and natural classes are well distinguished, the urban structure with waters or mountains is well preserved. Finally, we also discuss the impact of the sample receptive field and sample heterogeneity on LCZ classification performance, which provides a new idea for future studies of LCZ classification.