Multimodal Breast Cancer Diagnosis Based on Multi-level Fusion Network

With the widespread application of artificial intelligence technology, deep learning algorithms have been extensively applied to the diagnosis and screening of breast cancer. However, the classification of breast cancer with the data from a single modality is still not accurate enough to meet clinical needs. This paper proposes a Multimodal breast cancer diagnosis based on Multi-level fusion network which integrates pathological images, structured data and medical description text. Specifically, we first construct a fully connected graph to extract the node and graph level feature representation of pathological images with graph attention layers. Second, we use the BERT model to extract the text features from the medical records. At last, the features of the above three modal data are fused using a multimodal adaption gate (MAG) for diagnosis. Experimental results indicate that the proposed method obtains superior performance (accuracy 93.62%) to most baseline methods on PathoEMR dataset.