Real-time automated diagnosis of colorectal cancer invasion depth using a deep learning model with multimodal data (with video)

The optical diagnosis of colorectal cancer (CRC) invasion depth with white light (WL) and image-enhanced endoscopy (IEE) remains challenging. We aimed to construct and validate a 2-modal deep learning-based system, incorporated with both WL and IEE images (named Endo-CRC) in estimating the invasion depth of CRC.Samples were retrospectively obtained from 3 hospitals in China. We combined WL and IEE images into image pairs. Altogether, 337,278 image pairs from 268 noninvasive and superficial CRC and 181,934 image pairs from 82 deep CRC were used for training. A total of 296,644 and 4528 image pairs were used for internal and external tests and for comparison with endoscopists. Thirty-five videos were used for evaluating the real-time performance of the Endo-CRC system. Two deep learning models, solely using either WL (model W) or IEE images (model I), were constructed to compare with Endo-CRC.The accuracies of Endo-CRC in internal image tests with and without advanced CRC were 91.61% and 93.78%, respectively, and 88.65% in the external test, which did not include advanced CRC. In an endoscopist-machine competition, Endo-CRC achieved an expert comparable accuracy of 88.11% and the highest sensitivity compared with all endoscopists. In a video test, Endo-CRC achieved an accuracy of 100.00%. Compared with model W and model I, Endo-CRC had a higher accuracy (per image pair: 91.61% vs 88.27% compared with model I and 91.61% vs 81.32% compared with model W).The Endo-CRC system has great potential for assisting in CRC invasion depth diagnosis and may be well applied in clinical practice.