The Application of Deep Reinforcement Learning to Distributed Spectrum Access in Dynamic Heterogeneous Environments With Partial Observations

This papera ¹ investigates deep reinforcement learning (DRL) based on a Recurrent Neural Network (RNN) for Dynamic Spectrum Access (DSA) under partial observations, referred to as a Deep Recurrent Q-Network (DRQN). Specifically, we consider a scenario with multiple independent channels and multiple heterogeneous Primary Users (PUs). Two key challenges in our problem formulation are that we assume our DRQN node does not have any prior knowledge of the other nodes' behavior patterns and attempts to predict the future channel state based on previous observations. The goal of the DRQN is to learn a channel access strategy with a low collision rate but a high channel utilization rate. With proper definitions of the state, action and rewards, our extensive simulation results show that a DRQN-based approach can handle a variety of communication environments including dynamic environments. Further, our results show that the DRQN node is also able to cope with multi-rate and multi-agent scenarios. Importantly, we show the following benefits of using recurrent neural networks in DSA: (i) the ability to learn the optimal strategy in different environments under partial observations; (ii) robustness to imperfect observations and (iii) the ability to utilize multiple channels, and (iv) robustness in the presence of multiple agents. ¹ A parton of this work was presented at MILCOM 2018 in [1].