Due to variable and complex working conditions, rolling bearings are susceptible to compound faults. However, because a single fault plays a leading role, the weaker fault in the compound fault is difficult to monitor. If the weak fault in the compound fault can be found and identified, it is conducive to a more accurate judgment of the health state of the bearing. Several approaches have been proposed to detect compound bearing faults, but most of them are complex, inefficient and may require prior knowledge about fault characteristics such as fault period. In this article, the improved approach based on sparse filtering is proposed to detect the compound faults in rolling bearings. The proposed approach utilizes the wavelet decomposition method to divide the original vibration signal with compound faults into several segments in the frequency domain. All segmentations are employed to construct the Hankel matrix. Improved sparse filtering (ISF) is then employed to enhance the fault features by attenuating the noise. ISF performs autocorrelation on input samples (column of Hankel matrix) to improve the expressiveness of features. The penalty term is used to improve the performance of the sparse characteristic expressions of the weight matrix. The envelope spectral analysis is then finally used to detect the compound faults. The ability of sparse filtering to separate the multi-fault signal into different components is discussed in the simulation. Both simulation and experimental data with compound faults verify the effectiveness of the developed approach. The ability to distinguish different fault modes without prior knowledge of fault periods makes the proposed method advanced and suitable for compound fault diagnosis in rolling bearings. Compared with the existing methods, results show superior feature extraction performance of the proposed method.