Abstract Planetary gearbox generally operates under variable speed conditions in response to actual industrial production requirements, which is prone to cause single or compound fault. The planetary gearbox fault signal exhibits time-varying, weak, and complex characteristics under varying conditions, and common diagnostic approaches are difficult to effectively extract and reveal fault-related features. In addition, the computed order tracking requires speed signal as auxiliary calculation. Unfortunately, due to economic and installation space limitations, encoders and tachometers are not always available, which poses a huge challenge for extracting variable speed planetary gearbox fault. To tackle these challenges, a novel tacholess order tracking method based on improved adaptive chirp mode decomposition (IACMD) and adaptive mximum second-order cyclostationarity blind deconvolution (CYCBD) is proposed for diagnosing planetary gearbox single and composite fault in this contribution. Firstly, the IACMD algorithm is utilized to adaptively decompose the fault signal, which constructs a composite index (CI)-based signal mode selection and recombination scheme to perform optimal modal decomposition. Secondly, the instantaneous dominant meshing multiply (IDMM) trend line with obvious amplitude advantage is extracted from the time-frequency representation of the original fault signal, and then converted to the reference shaft rotation frequency to perform angle domain resampling on sensitive model with the highest CI value. Thirdly, the adaptive CYCBD is adopted to deconvolute the angular domain sensitive modal to heighten the weak fault signatures. Finally, envelope spectrum of improved signal is utilized to identify prominent fault characteristic orders and ascertain failure type. Both numerical simulations and practical engineering data in different planetary gearbox failure cases have been thoroughly examined to demonstrate the correctness and feasibility of the proposed approach. Moreover, compared with some existing technologies demonstrates the superiority of the proposed approach.