3MO Multimodal data fusion for improved risk stratification of breast cancer with multi-task 3D deep learning model: A multicenter study

Timely intervention and improved prognosis for breast cancer patients rely on early metastasis risk detection and accurate treatment predictions. This study aims to the amalgamation of artificial intelligence innovation and medical research by developing a novel multi-task 3D deep learning model with MRI-based multimodal data fusion. This pioneering multicenter study involves 1,244 non-metastatic breast cancer patients, who were assigned into the training cohort (n = 456), internal validation cohort (n = 113), external testing cohort 1 (n = 432), and external testing cohort 2 (n = 198). An innovative multimodal approach integrating clinicopathological data with deep learning MRI insights yielded the multi-task 3D deep learning model (3D-MMR-model), which was developed for tumor segmentation and disease-free survival (DFS) prediction. The efficacy was demonstrated through tumor segmentation accuracy metrics and DFS prediction AUC values. Visualization techniques provided insight into decision-making processes, correlating model predictions with the tumor microenvironment. The 3D-MMR-model demonstrated a high degree of predictive accuracy and significant boost for DFS. The AUC for 4-year DFS prediction escalated to 0.98, 0.97, 0.90, and 0.93 within the training cohort, internal validation cohort, external testing cohort 1, and external testing cohort 2, respectively. Our multimodal model showcased significant distinctions in DFS between patients with high versus low risk scores (All P < 0.001). Moreover, a decision curve analysis underscored that the multimodal model yielded a superior net benefit across a broad range of threshold probabilities within all cohorts, which implies the multimodal model adds substantial clinical value to early DFS prediction. Furthermore, patients in the high-risk group displayed concentrated hotspots in regions near or distant from the tumor and revealed an elevated presence of antigen-presenting cells. This study introduces a transformative approach to breast cancer prognosis, amalgamating imaging and clinical data for enhanced predictive accuracy, thus holding promise for personalized treatment strategies.