A hybrid attention multi-scale fusion network for real-time semantic segmentation

Abstract In semantic segmentation research, spatial information and receptive fields are essential. However, currently, most algorithms focus on acquiring semantic information and lose a significant amount of spatial information, leading to a significant decrease in accuracy despite improving real-time inference speed. This paper proposes a new method to address this issue. Specifically, we have designed a new module (HFRM) that combines channel attention and spatial attention to retrieve the spatial information lost during downsampling and enhance object classification accuracy. Regarding fusing spatial and semantic information, we have designed a new module (HFFM) to merge features of two different levels more effectively and capture a larger receptive field through an attention mechanism. Additionally, edge detection methods have been incorporated to enhance the extraction of boundary information. Experimental results demonstrate that for an input size of 512 × 1024, our proposed method achieves 73.6% mIoU at 176 frames per second (FPS) on the Cityscapes dataset and 70.0% mIoU at 146 FPS on Camvid. Compared to existing networks, our Model achieves faster inference speed while maintaining accuracy, enhancing its practicality.