The application of machine learning based energy management strategy in multi-mode plug-in hybrid electric vehicle, part I: Twin Delayed Deep Deterministic Policy Gradient algorithm design for hybrid mode

As the performance of Energy Management Strategy (EMS) is crucial for the energy efficiency of Hybrid Electric Vehicles (HEVs), a Deep Reinforcement Learning (DRL)-based algorithm, namely Twin Delayed Deep Deterministic Policy Gradient (TD3), is adopted to design EMS for the power Charge-Sustained (CS) stage of a multi-mode plug-in Hybrid Electric Vehicle (HEV). In addition, EMS is improved by combining the actor-network of TD3 with Gumbel-Softmax to realize mode selection and torque distribution simultaneously, which is a discrete (mode)-continuous (engine speed) hybrid action space and not applicable in original TD3. To reduce the unreasonable exploration of agents in discrete action, a rule-based mode control mechanism (RBMCM) is designed and involved in EMS. The improved algorithm speeds up the learning process and achieves better fuel economy. Simulation results show that the gap between the proposed strategy and the benchmark dynamic programming (DP) is reduced to 2.55% in the selected training cycle. Regarding the unknown testing cycles, the fuel economy of agents trained by the improved method overperforms traditional DRL-based EMS when it reaches more than 90% of the DP-based benchmarking. In conclusion, the proposed method provides a theoretical foundation for the solution of the hybrid space optimization problem for hybrid systems.