Multi-agent deep reinforcement learning for resilience-driven routing and scheduling of mobile energy storage systems

Extreme events are featured by high impact and low probability, which can cause severe damage to power systems. There has been much research focused on resilience-driven operational problems incorporating mobile energy storage systems (MESSs) routing and scheduling due to its mobility and flexibility. However, existing literature focuses on model-based optimization approaches to implement the routing process of MESSs, which can be time consuming and raise privacy issues since the requirement for global information of both power and transportation networks. Furthermore, a real-time automatic control scheme of MESSs has become a challenging task due to the system high variability. As such, this paper develops a model-free real-time multi-agent deep reinforcement learning approach featuring parameterized double deep Q-networks to reformulate the coordination effect of MESSs routing and scheduling process as a Partially Observable Markov Game, which is capable of capturing a hybrid policy including both discrete and continuous actions. A coupled transportation network and linearized AC-OPF algorithm are realized as the environment, while the internal uncertainties associated with renewable energy sources, load profiles, line outages, and traffic volumes are incorporated into the proposed data-driven approach through learning procedure. Extensive case studies including both 6-bus and 33-bus power networks are developed to evaluate the effectiveness of the proposed approach. Specifically, a detailed comparison between different multi-agent reinforcement learning and model-based optimization approaches is conducted to present the superior performance of the proposed approach.