A Novel Effectiveness Assessment Framework for Neoadjuvant Chemoradiotherapy of Locally Advanced Rectal Cancer Based on Multi-modal Intelligence

Neoadjuvant chemoradiotherapy (nCRT) is the stan-dard treatment for locally advanced rectal cancer (LARC). With the development of artificial intelligence, an increasing number of studies have begun to explore its application in cancer treatment prediction. However, the prior methods exhibit considerable variability even with slight modifications to the input data, which could potentially undermine the reliability of the results. In this paper, we proposed RP-Net, a novel multi-modal fusion-based framework that combines feature information from magnetic resonance imaging (MRI) and whole slide images (WSI), establishing a relationship to map the therapeutic effectiveness of nCRT for LARC. We investigated the relationship of the tumour region and its periphery tissues, and demonstrated the validity of the proposed framework that involving 11 different combinations of modalities. The experimental results revealed that it has achieved higher prediction accuracy compared to the four intra-categories single-modal combinations and outperformed the two intra-categories multi-modal combinations. When compared to the other four inter-categories multi-modal combinations, the fusion features get accuracy of 2 % ~ 6% improvement respectively.