Multimodal microscopic imaging with deep learning for highly effective diagnosis of breast cancer

Accurate and rapid diagnosis of breast cancer is particularly important since it becomes the primary malignant tumor threatening women's health. Although pathological diagnosis with H&E staining section is the gold standard in tumor diagnosis, it is hard to meet quick intraoperative inspection because of its complex procedures and hysteresis. This study adopts multimodal microscopic imaging technology (bright-field imaging, auto-fluorescence imaging and orthogonal polarization imaging) combined with deep learning to achieve rapid intelligent diagnosis of breast cancer by getting the rich information of tissue morphology, content and structure of collagen in tissue slices. By using both fusion classification models of multimodal images at pixel level and decision level, the AUC score and accuracy is 0.9366 and 89.01% for the pixel fusion classification, as well as 0.9421 and 87.53% for the decision fusion classification, respectively. The results suggest that the multimodal microscopic imaging technique proposed in this study has unique advantages in accuracy, speed and feasibility in clinic for breast cancer diagnosis, and has strong clinical potential and application prospects when combined with deep learning.