Deep-Reinforcement-Learning-Based Age-of-Information-Aware Low-Power Active Queue Management for IoT Sensor Networks

As the number of Internet of Things (IoT) sensors increases and their deployment becomes denser, power management for IoT sensor networks becomes more important. In most IoT sensor networks, one cluster head (CH) collects data from a large number of sensors and forwards them to backbone networks. Thus, managing CH's queue condition is crucial in order to extend the network's lifespan or satisfy quality of service (QoS) requirements. Meanwhile, reducing age of information (AoI), a metric describing how fresh information is, has become one of the most important metrics, from simple data such as temperature and humidity to more complex data that must be timely, such as vehicle information and road dynamics. However, it is shown that AoI may heavily fluctuate depending on the medium access control protocol. In this paper, we propose a deep reinforcement learning-based AoI-aware low-power active queue management for IoT sensor networks. To this end, we formulate a Markov decision process model in which the CH can select one of the actions including forward, flush, or leave buffered data from associated cluster member nodes. Extensive simulations show that compared with traditional queue management methods, our queue management method can reduce the power consumption of CH while trying not to exceed the AoI value threshold, thereby enabling IoT sensor networks to be stable while ensuring satisfactory QoS.