During metal cutting, chatter is prone to the effects of poor surface quality and tool wear. Therefore, chatter detection is becoming more and more important. The current hot methods are effective, but they also have limitations, such as the interference of human experience on the results, the need to label the data, and it takes a long time. This paper proposes an unsupervised milling chatter detection method based on a large number of unlabeled dynamic signals. The method does not depend on processing parameters and environment, does not require labels, and has strong stability. Based on auto-encode, each segment of the signal is compressed into two dimensions, and the feasibility of the reconstruction scheme is verified by numerical analysis. In the normalization algorithm, the similarity between the raw signal and the reconstructed signal is the highest, and the reconstruction effect is the best. Then, the compressed signals are clustered based on a hybrid clustering method combining GMM and K-means. Under the six evaluation indicators, compared with GMM, the clustering results of this scheme have been significantly improved. The evaluation metrics show that GMM-K-means is not only more stable but also has better result compared to K-means in chatter detection. The results show that the proposed method outperforms GMM and K-means in all six typical unsupervised evaluation metrics, and can detect chatter effectively.