Owing to the effect of gear flexibility, the extended tooth contact occurs, which is a phenomenon that the incoming tooth pair enters contact earlier than the theoretical start of contact, and the outgoing tooth pair leaves contact later than the theoretical end of contact. Considering the effects of the extended tooth contact and tooth root crack on the time-varying mesh stiffness (TVMS), a finite element (FE) model of a spur gear pair in mesh is established by ANSYS software. TVMS under different crack depths at constant rated torque (60 Nm) are calculated based on the FE model. Then, a FE model of a geared rotor system is developed by MATLAB software. Frequency-domain features, statistical features (Kurtosis and RMS) and instantaneous energies based on empirical mode decomposition (EMD) under different crack depths are calculated at 1000 rev/min and the corresponding measured results are also performed by model experiment. The results show that considering the effect of extended tooth contact, TVMS of crack gear pair becomes smooth from double-tooth engagement to single-tooth engagement; the gear crack has reduced the gear body rigidity, which leads to the reduction of stiffness of the healthy teeth and has little effect on the vibration response. The instantaneous energy can be chosen as a distinguishing indicator to qualitatively diagnose the gear cracks with different levels.