Ulcerative Colitis, LAIR1 and TOX2 Expression, and Colorectal Cancer Deep Learning Image Classification Using Convolutional Neural Networks

Background: Ulcerative colitis is a chronic inflammatory bowel disease of the colon mucosa associated with a higher risk of colorectal cancer. Objective: This study classified hematoxylin and eosin (H&E) histological images of ulcerative colitis, normal colon, and colorectal cancer using artificial intelligence (deep learning). Methods: A convolutional neural network (CNN) was designed and trained to classify the three types of diagnosis, including 35 cases of ulcerative colitis (n = 9281 patches), 21 colon control (n = 12,246), and 18 colorectal cancer (n = 63,725). The data were partitioned into training (70%) and validation sets (10%) for training the network, and a test set (20%) to test the performance on the new data. The CNNs included transfer learning from ResNet-18, and a comparison with other CNN models was performed. Explainable artificial intelligence for computer vision was used with the Grad-CAM technique, and additional LAIR1 and TOX2 immunohistochemistry was performed in ulcerative colitis to analyze the immune microenvironment. Results: Conventional clinicopathological analysis showed that steroid-requiring ulcerative colitis was characterized by higher endoscopic Baron and histologic Geboes scores and LAIR1 expression in the lamina propria, but lower TOX2 expression in isolated lymphoid follicles (all p values < 0.05) compared to mesalazine-responsive ulcerative colitis. The CNN classification accuracy was 99.1% for ulcerative colitis, 99.8% for colorectal cancer, and 99.1% for colon control. The Grad-CAM heatmap confirmed which regions of the images were the most important. The CNNs also differentiated between steroid-requiring and mesalazine-responsive ulcerative colitis based on H&E, LAIR1, and TOX2 staining. Additional classification of 10 new cases of colorectal cancer (adenocarcinoma) were correctly classified. Conclusions: CNNs are especially suited for image classification in conditions such as ulcerative colitis and colorectal cancer; LAIR1 and TOX2 are relevant immuno-oncology markers in ulcerative colitis.