Federated reinforcement learning for smart building joint peer-to-peer energy and carbon allowance trading

The multi-energy system (MES), which is regarded as an optimum solution to a high-efficiency, green energy system and a crucial shift towards the future low-carbon energy system, has attracted great attention at the district building level. However, the current exploration of flexible MES operation has been hampered by (1) the increasing penetration of renewable energies and the complicated operation of coupling multi-energy sectors; (2) the privacy concern in the decentralization of the energy system; and (3) the lack of integration of the energy market and carbon emission trading scheme. To address the aforementioned challenges, this paper proposes a joint peer-to-peer energy and carbon allowance trading mechanism for a building community, and then models it as a multi-agent reinforcement learning (MARL) paradigm. In this setting, the flexibility of building local trading and the decarbonization of building energy management can both be fully utilized. To stabilize the training performance, an abstract critic network capturing system dynamics is introduced based on a deep deterministic policy gradient method. The technique of federated learning (FL) is also applied to speed up the training and safeguard the private information of each building in the community. Empirical results on a real-world test case evaluate its superior performance in terms of achieving both economic and environmental benefits, resulting in 5.87% and 8.02% lower total energy and environment costs than the two baseline mechanisms of peer-to-grid energy trading and peer-to-peer energy trading, respectively.