Automated Quantification of HER2 Amplification Levels Using Deep Learning

HER2 assessment is necessary for patient selection in anti-HER2 targeted treatment. However, manual assessment of HER2 amplification is time-costly, labor-intensive, highly subjective and error-prone. Challenges in HER2 analysis in fluorescence in situ hybridization (FISH) and dual in situ hybridization (DISH) images include unclear and blurry cell boundaries, large variations in cell shapes and signals, overlapping and clustered cells and sparse label issues with manual annotations only on cells with high confidences, producing subjective assessment scores according to the individual choices on cell selection. To address the above-mentioned issues, we have developed a soft-sampling cascade deep learning model and a signal detection model in quantifying CEN17 and HER2 of cells to assist assessment of HER2 amplification status for patient selection of HER2 targeting therapy to breast cancer. In evaluation with two different kinds of clinical datasets, including a FISH data set and a DISH data set, the proposed method achieves high accuracy, recall and F1-score for both datasets in instance segmentation of HER2 related cells that must contain both CEN17 and HER2 signals. Moreover, the proposed method is demonstrated to significantly outperform seven state of the art recently published deep learning methods, including contour proposal network (CPN), soft label-based FCN (SL-FCN), modified fully convolutional network (M-FCN), bilayer convolutional network (BCNet), SOLOv2, Cascade R-CNN and DeepLabv3+ with three different backbones (p ≤ 0.01). Clinically, anti-HER2 therapy can also be applied to gastric cancer patients. We applied the developed model to assist in HER2 DISH amplification assessment for gastric cancer patients, and it also showed promising predictive results (accuracy 97.67 ±1.46%, precision 96.15 ±5.82%, respectively).