Deep fusion of human-machine knowledge with attention mechanism for breast cancer diagnosis

Breast cancer is a common disease worldwide that poses a significant threat to the health of women. Many researchers have developed computer-aided diagnosis (CAD) systems to help clinicians identify breast cancer. However, the existing methods ignore the combination of image information and human clinical description in CAD systems. In this paper, we propose a novel breast tumor classification system based on spatial attention and cross-semantic human–machine knowledge fusion. Our system pairs the ultrasound image and human scoring data as the input of the network and maps them to the same feature space. Then a human–machine knowledge aggregation network based on channel attention is proposed to fuse features from images and human descriptions to classify breast tumors. In the image feature extraction process, we propose a spatial attention convolution neural network to automatically focus on the key regions of the image related to classification. We have conducted cross-validation experiments, and the comparative results have shown that our method can effectively improve classification performance and achieve the highest value in five evaluation metrics.