MFH‐Net: A Hybrid CNN‐Transformer Network Based Multi‐Scale Fusion for Medical Image Segmentation

ABSTRACT In recent years, U‐Net and its variants have gained widespread use in medical image segmentation. One key aspect of U‐Net's design is the skip connection, facilitating the retention of detailed information and leading to finer segmentation results. However, existing research often concentrates on enhancing either the encoder or decoder, neglecting the semantic gap between them, and resulting in suboptimal model performance. In response, we introduce Multi‐Scale Fusion module aimed at enhancing the original skip connections and addressing the semantic gap. Our approach fully incorporates the correlation between outputs from adjacent encoder layers and facilitates bidirectional information exchange across multiple layers. Additionally, we introduce Channel Relation Perception module to guide the fused multi‐scale information for efficient connection with decoder features. These two modules collectively bridge the semantic gap by capturing spatial and channel dependencies in the features, contributing to accurate medical image segmentation. Building upon these innovations, we propose a novel network called MFH‐Net. On three publicly available datasets, ISIC2016, ISIC2017, and Kvasir‐SEG, we perform a comprehensive evaluation of the network. The experimental results show that MFH‐Net exhibits higher segmentation accuracy in comparison with other competing methods. Importantly, the modules we have devised can be seamlessly incorporated into various networks, such as U‐Net and its variants, offering a potential avenue for further improving model performance.