With the penetration of renewable energy sources (RES) increasing rapidly, cascading failures become more complex due to the uncertainty and vulnerability of RES. This paper presents an online cascading failure searching method based on gradient boosting decision tree (GBDT) for the hybrid AC/DC system with high penetration of wind power. First, the cascading failure risk index is established, which considers the impact, probability, and loss of failures. Then, a cascading failure searching based on Monte Carlo tree search and dynamic simulation is performed to acquire the high-risk cascading failures, which are utilized as the training samples. Finally, GBDT is deployed to construct the relationship between operating conditions and failure propagation directions. Online cascading failure searching is realized by combining GBDT failure estimation and dynamic failure simulation. Simulation results of the modified New England 39-bus test system demonstrate that the proposed method can fast and accurately online screen the high-risk cascading failures considering the uncertainty of wind power.