Path planning for unmanned surface vehicle based on improved Q-Learning algorithm

Path planning is a key factor for the unmanned surface vehicle (USV) to achieve efficient navigation. In this paper, to solve the global path planning and obstacle avoidance problems for the USV, an improved Q-Learning algorithm called neural network smoothing and fast Q-Learning (NSFQ) is proposed. Three main improvement parts are composed of the proposed algorithm. Firstly, the radial basis function (RBF) neural network is combined with the Q-Learning algorithm to approximate the action value function Q, which improves the convergence speed of the Q-Learning algorithm. Secondly, to ensure that the planned path conforms to the maneuvering characteristics of the USV, the heading angle, motion characteristics, ship length, and safety of the USV are taken into account by the proposed algorithm. Based on these factors, the action space and reward function are optimized, the state space is reconstructed, and the safety threshold is proposed. Finally, a third-order Bezier curve is used to smooth the initial path, so that the USV can maintain its heading stability during navigation. Based on simulation results, the proposed NSFQ algorithm outperforms the A* and RRT algorithms in terms of evaluation indicators such as heading angle, angular velocity, path length, sailing time, and path smoothness.