Graph clustering has received significant attention in recent years due to the breakthrough of graph neural networks (GNNs). However, GNNs frequently assume strong data homophily, which is not true in many real-world applications. Furthermore, practical graphs are typically noisy and sparse, which inevitably degrades the clustering performance. To this end, we propose a novel Contrastive Graph Clustering (CGC) method with adaptive filter framework. We first design an adaptive filter that can automatically learn a suitable filter for different data, mining holistic information beyond low-frequency components and encoding topology structure information into features. Afterward, we learn a refined graph based on a graph-level contrastive mechanism, which further boosts graph discriminability. Extensive experiments show that the proposed CGC method achieves significant improvement over state-of-the-art methods on several benchmark datasets. In particular, our simple method, which does not employ neural networks, outperforms many deep learning approaches.