Machine learning algorithm and deep neural networks identified a novel subtype in hepatocellular carcinoma

BACKGROUND: Hepatocellular carcinoma (HCC) is one of the most common malignant tumors. Due to the lack of specific characteristics in the early stage of the disease, patients are usually diagnosed in the advanced stage of disease progression. OBJECTIVE: This study used machine learning algorithms to identify key genes in the progression of hepatocellular carcinoma and constructed a prediction model to predict the survival risk of HCC patients. METHODS: The transcriptome data and clinical information were downloaded from The Cancer Genome Atlas (TCGA) and Gene Expression Omnibus (GEO). The differential expression analysis and COX proportional-hazards model participated in the identification of survival-related genes. K-Means, Random forests, and LASSO regression are involved in identifying novel subtypes of HCC and screening key genes. The prediction model was constructed by deep neural networks (DNN), and Gene Set Enrichment Analysis (GSEA) reveals the metabolic pathways where key genes are located. RESULTS: Two subtypes were identified with significantly different survival rates (p< 0.0001, AUC = 0.720) and 17 key genes associated with the subtypes. The accuracy rate of the deep neural network prediction model is greater than 93.3%. The GSEA analysis found that the survival-related genes were significantly enriched in hallmark gene sets in the MSigDB database. CONCLUSIONS: In this study, we used machine learning algorithms to screen out 17 genes related to the survival risk of HCC patients, and trained a DNN model based on them to predict the survival risk of HCC patients. The genes that make up the model are all key genes that affect the formation and development of cancer.