An Improved Multi-Objective Deep Reinforcement Learning Algorithm Based on Envelope Update

Multi-objective reinforcement learning (MORL) aims to uniformly approximate the Pareto frontier in multi-objective decision-making problems, which suffers from insufficient exploration and unstable convergence. We propose a multi-objective deep reinforcement learning algorithm (envelope with dueling structure, Noisynet, and soft update (EDNs)) to improve the ability of the agent to learn optimal multi-objective strategies. Firstly, the EDNs algorithm uses neural networks to approximate the value function and update the parameters based on the convex envelope of the solution boundary. Then, the DQN structure is replaced with the dueling structure, and the state value function is split into the dominance function and value function to make it converge faster. Secondly, the Noisynet method is used to add exploration noise to the neural network parameters to make the agent have a more efficient exploration ability. Finally, the soft update method updates the target network parameters to stabilize the training procedure. We use the DST environment as a case study, and the experimental results show that the EDNs algorithm has better stability and exploration capability than the EMODRL algorithm. In 1000 episodes, the EDNs algorithm improved the coverage by 5.39% and reduced the adaptation error by 36.87%.