Application of Deep Learning Models to Improve Ulcerative Colitis Endoscopic Disease Activity Scoring Under Multiple Scoring Systems

Abstract Background and Aims Lack of clinical validation and inter-observer variability are two limitations of endoscopic assessment and scoring of disease severity in patients with ulcerative colitis [UC]. We developed a deep learning [DL] model to improve, accelerate and automate UC detection, and predict the Mayo Endoscopic Subscore [MES] and the Ulcerative Colitis Endoscopic Index of Severity [UCEIS]. Methods A total of 134 prospective videos [1550 030 frames] were collected and those with poor quality were excluded. The frames were labelled by experts based on MES and UCEIS scores. The scored frames were used to create a preprocessing pipeline and train multiple convolutional neural networks [CNNs] with proprietary algorithms in order to filter, detect and assess all frames. These frames served as the input for the DL model, with the output being continuous scores for MES and UCEIS [and its components]. A graphical user interface was developed to support both labelling video sections and displaying the predicted disease severity assessment by the artificial intelligence from endoscopic recordings. Results Mean absolute error [MAE] and mean bias were used to evaluate the distance of the continuous model’s predictions from ground truth, and its possible tendency to over/under-predict were excellent for MES and UCEIS. The quadratic weighted kappa used to compare the inter-rater agreement between experts’ labels and the model’s predictions showed strong agreement [0.87, 0.88 at frame-level, 0.88, 0.90 at section-level and 0.90, 0.78 at video-level, for MES and UCEIS, respectively]. Conclusions We present the first fully automated tool that improves the accuracy of the MES and UCEIS, reduces the time between video collection and review, and improves subsequent quality assurance and scoring.