Multi-Task Learning with Context-Oriented Self-Attention for Breast Ultrasound Image Classification and Segmentation

Breast cancer is a great threat to women’s health. Automatic analysis of Breast UltraSound (BUS) images can help radiologists make more accurate and efficient diagnoses of breast cancer. We propose a Multi-Task Learning Network with Context-Oriented Self-Attention (MTL-COSA) module to automatically and simultaneously segment tumors and classify them as benign or malignant. The COSA module incorporates prior medical knowledge to guide the network to learn contextual relationships for better feature representations in BUS images. Extensive cross-validation experiments are conducted on two public datasets to evaluate the performance of MTL-COSA and several state-of-the-art methods. MTL-COSA achieves the best classification results and second-best segmentation results compared with deep learning-based methods (5 classification methods and 3 segmentation methods).