Joint Routing and Scheduling Optimization in Time-Sensitive Networks Using Graph-Convolutional-Network-Based Deep Reinforcement Learning

The growing number of Internet of Things (IoT) devices brings enormous time-sensitive applications, which require real-time transmission to effectuate communication services. The ultrareliable and low-latency communication (URLLC) scenario in the fifth generation (5G) has played a critical role in supporting services with delay-sensitive properties. Time-sensitive networking (TSN) has been widely considered as a promising paradigm for enabling the deterministic transmission guarantees for 5G. However, TSN is a hybrid traffic system with time-sensitive traffic and best effort traffic, which require effective routing and scheduling to provide a deterministic and bounded delay. While joint optimization of time-sensitive and non-time-sensitive traffic greatly increases the solution space and brings a significant challenge to obtain solutions. Therefore, this article proposes a graph convolutional network-based deep reinforcement learning (GCN-based DRL) solution for the joint optimization problem in practical communication scenarios. The GCN is integrated into deep reinforcement learning (DRL) to obtain the network's spatial dependence and elevate the generalization performance of the proposed method. Specifically, the GCN adopts the first-order Chebyshev polynomial to approximate the graph convolution kernel, which reduces the complexity of the algorithm and improves the feasibility for the joint optimization task. Furthermore, priority experience replay is employed to accelerate the convergence speed of the model training process. Numerical simulations demonstrate that the proposed GCN-based DRL algorithm has good convergence and outperforms the benchmark methods in terms of the average end-to-end delay.