Urban Traffic Signal Control with Reinforcement Learning from Demonstration Data

Reinforcement learning has been applied to various decision-making tasks and has achieved high profile successes. More and more studies have proposed to use reinforcement learning (RL) for traffic signal control to improve transportation efficiency. However, these methods suffer from a major exploration problem, and their performance is particularly poor. And even fail to quickly converge during the initial stage when interacting with the environment. To overcome this problem, we propose an RL model for traffic signal control based on demonstration data, which provides prior expert knowledge before RL model training. The demonstrations are collected from the classic method self-organizing traffic light (SOTL). It not only serves as expert knowledge but also explores and improves the entire decision-making system. Specifically, we use small demonstration data sets to pre-train the Ape-X Deep Q-learning Network (DQ N) for traffic signal control. When training a RL model from scratch, we often need a lot of data and time to learn a better initialization. Our approach is dedicated to making the RL algorithm converge quickly and accelerating the pace of learning. Extensive experiments on three urban datasets confirm that our method performs better with faster convergence and least travel time than the current RL-based methods by an average of 23.9%, 23.8%, 11.6%