A synchronous multi-agent reinforcement learning framework for UVMS grasping

The lightweight design of underwater vehicle-manipulator systems (UVMS) enhances task flexibility and adaptability but also adds complexity to controller design. With the lightweight underwater vehicle, the underwater manipulator accounts for a larger proportion of the total weight. The dynamic coupling between the underwater manipulator and the underwater vehicle becomes more pronounced, resulting in reduced accuracy in end-effector operations. Recent advancements in artificial intelligence offer a potential solution. In this paper, we propose a multi-agent reinforcement learning framework for UVMS control. Two agents are trained to control the underwater manipulator and vehicle, respectively, in order to address the challenges caused by dynamic coupling. The actor–critic network is employed for both agents based on the Proximal Policy Optimization (PPO) algorithm. A synchronous training method for multi-agents is embedded to improve training performance. Simulation results demonstrate that the proposed method achieves a higher success rate of 95% in object-grasping experiments compared to both the single-agent learning method and the asynchronous training method. Furthermore, strategies aimed at bridging the gap between simulation and real-world environments facilitate successful grasping by the physical UVMS, even though there is a decrease in the success rate during pool experiments.