Constrained Hierarchical Hybrid Q-Network for Energy Management of HEVs

Energy management strategy (EMS) is a crucial technology for ensuring the fuel efficiency of hybrid electric vehicles (HEVs). However, the complex discrete-continuous hybrid action space and physical constraints in the powertrain of HEVs present a challenge for developing high-performance EMSs based on deep reinforcement learning (DRL). This paper proposes a constrained hierarchical hybrid Q-network (CHHQN) algorithm, based on which a two-level EMS framework is built for direct learning within the hybrid action space, encompassing both torque distribution and gear-shifting strategies. To ensure that critical metrics like the battery's state of charge are not violated, the designed EMS introduces an additional safety layer to correct the agent's actions. The CHHQN-based EMS exhibits only a 3.73% difference to dynamic programming in fuel consumption. Comprehensive comparisons with other typical DRL-based methods, e.g., deep deterministic policy gradients and deep Q-network, demonstrate a considerable fuel economy improvement. The effectiveness of the proposed method is validated through a hardware-in-loop test.