Development and validation of an artificial intelligence‐based system for predicting colorectal cancer invasion depth using multi‐modal data

Objectives Accurate endoscopic optical prediction of the depth of cancer invasion is critical for guiding an optimal treatment approach of large sessile colorectal polyps but was hindered by insufficient endoscopists expertise and inter‐observer variability. We aimed to construct a clinically applicable artificial intelligence (AI) system for the identification of presence of cancer invasion in large sessile colorectal polyps. Methods A deep learning‐based colorectal cancer invasion calculation (CCIC) system was constructed. Multi‐modal data including clinical information, white light (WL) and image‐enhanced endoscopy (IEE) were included for training. The system was trained using 339 lesions and tested on 198 lesions across three hospitals. Man–machine contest, reader study and video validation were further conducted to evaluate the performance of CCIC. Results The overall accuracy of CCIC system using image and video validation was 90.4% and 89.7%, respectively. In comparison with 14 endoscopists, the accuracy of CCIC was comparable with expert endoscopists but superior to all the participating senior and junior endoscopists in both image and video validation set. With CCIC augmentation, the average accuracy of junior endoscopists improved significantly from 75.4% to 85.3% ( P = 0.002). Conclusions This deep learning‐based CCIC system may play an important role in predicting the depth of cancer invasion in colorectal polyps, thus determining treatment strategies for these large sessile colorectal polyps.