Multi-component extraction is an available method for vibration signal analysis of rotary machinery, so a novel method of rubbing fault diagnosis based on variational mode decomposition (VMD) is proposed. VMD is a newly developed technique for adaptive signal decomposition, which can non-recursively decompose a multi-component signal into a number of quasi-orthogonal intrinsic mode functions. The equivalent filtering characteristics of VMD are investigated, and the behavior of wavelet packet-like expansion is first found based on fractional Gaussian noise via numerical simulations. VMD is then applied to detect multiple rubbing-caused signatures for rotor–stator fault diagnosis via numerical simulated response signal and practical vibration signal. A comparison has also been conducted to evaluate the effectiveness of identifying the rubbing-caused signatures by using VMD, empirical wavelet transform (EWT), EEMD and EMD. The analysis results of the rubbing signals show that the multiple features can be better extracted with the VMD, simultaneously.