How GNNs Facilitate CNNs in Mining Geometric Information from Large-Scale Medical Images

Gigapixel medical images are a rich source of information containing both morphological textures and spatial information. However, existing deep learning solutions primarily rely on convolutional neural networks (CNNs) for global pixel-level analysis, ignoring the underlying local geometric structure. Since the topological structure in medical images is closely related to tumor evolution, graphs can be utilized to characterize it. To obtain a more comprehensive representation for downstream analysis, a fusion framework is proposed to enhance the global image-level representation captured by CNNs with the geometry of cell-level spatial information learned by graph neural networks (GNN). Two fusion strategies have been developed: one with MLP, which is simple but efficient through fine-tuning, and the other with TRANSFORMER, which excels in fusing multiple networks. The proposed fusion strategies have been evaluated on histology datasets from large patient cohorts of colorectal and gastric cancers for three biomarker prediction tasks. Both models outperform plain CNNs or GNNs, achieving a consistent AUC improvement of more than 5% on various network backbones. Importantly, the experimental results demonstrate the necessity of combining image-level morphological features with cell spatial relations in medical image analysis. Codes are available at https://github.com/yiqings/HEGnnEnhanceCnn.