ScCAEs: deep clustering of single-cell RNA-seq via convolutional autoencoder embedding and soft K-means

Clustering and cell type classification are a vital step of analyzing scRNA-seq data to reveal the complexity of the tissue (e.g. the number of cell types and the transcription characteristics of the respective cell type). Recently, deep learning-based single-cell clustering algorithms become popular since they integrate the dimensionality reduction with clustering. But these methods still have unstable clustering effects for the scRNA-seq datasets with high dropouts or noise. In this study, a novel single-cell RNA-seq deep embedding clustering via convolutional autoencoder embedding and soft K-means (scCAEs) is proposed by simultaneously learning the feature representation and clustering. It integrates the deep learning with convolutional autoencoder to characterize scRNA-seq data and proposes a regularized soft K-means algorithm to cluster cell populations in a learned latent space. Next, a novel constraint is introduced to the clustering objective function to iteratively optimize the clustering results, and more importantly, it is theoretically proved that this objective function optimization ensures the convergence. Moreover, it adds the reconstruction loss to the objective function combining the dimensionality reduction with clustering to find a more suitable embedding space for clustering. The proposed method is validated on a variety of datasets, in which the number of clusters in the mentioned datasets ranges from 4 to 46, and the number of cells ranges from 90 to 30 302. The experimental results show that scCAEs is superior to other state-of-the-art methods on the mentioned datasets, and it also keeps the satisfying compatibility and robustness. In addition, for single-cell datasets with the batch effects, scCAEs can ensure the cell separation while removing batch effects.