Memory-Net: Coupling feature maps extraction and hierarchical feature maps reuse for efficient and effective PET/CT multi-modality image-based tumor segmentation

Dense connection (DC), which densely reuses shallow feature maps to boost feature propagation, has been widely used in medical image segmentation models. However, feature aggregation of DC can be redundant, resulting in an expensive computational cost for resource-constrained platforms. To address this problem, we proposed a gated convolutional unit (GCU)-based feature forwarding method. GCU is a modified convolutional long short-term memory (Conv-LSTM) unit. First, we present a gated convolutional module (GCM) by hierarchically stacking GCUs and combining it with UNet architecture to create Memory-Net, wherein the hidden states of GCUs serve as the feature maps and the cell memories throughout the GCM form an information highway, enabling the model to efficiently and selectively aggregate useful information to boost feature propagation. We further combine the GCU with hyperdense connection (HDC) to propose a hyper-gated convolutional unit (HGCU) and develop a novel GCU and HGCU based multi-branch encoder (GCU-HGCU-Encoder), wherein the cell memory of the encoding branch for a specific input modality is not only used within the branch but also across all encoding branches, resulting in more efficient and effective multi-modality information fusion for Memory-Net. For improved segmentation, we introduced a recurrent-dense-Siamese decoder (RDS-Decoder) to create Memory-HDRDS-UNet, which is the final proposal of Memory-Net, by combining a GCU-HGCU-Encoder, a simple RDS-Decoder, and skip layers. We validated the superiority of Memory-Net on PET/CT volumes with lymphomas, and it achieved an average Dice score of 0.8690 and an average sensitivity of 0.9616, outperforming the state-of-the-art methods with a much lower computational cost.