Using deep learning to optimise road network traffic management

Abstract

An increasing demand on transportation in cities and urban areas is leading to more traffic congestion and consequently an increase in transportation costs and environmental pollution. In order to minimise this congestion we explore methods that focus on making more efficient use of existing road infrastructure. In particular, we focus on using deep learning techniques that are able to optimally control traffic lights at intersections with the aims of minimising traffic congestion and maximising traffic flow. In the first phase of our research project we develop a novel interactive simulation environment using the Unity Game Engine which model three different road network scenarios of incremental complexity. This simulation environment is designed in such a way that human users can easily change the colours of the traffic lights within the scenario. We use this simulation environment to record a data set of human experts interacting with simulation environment in an optimal manner to minimise traffic congestion and maximise traffic flow. We then develop two different sets of intelligent traffic light controllers which make use of Machine Learning to achieve the aims set out in this project. The first set of intelligent traffic light controllers consist of two agents which make use of deep Reinforcement Learning (RL) techniques to learn how to optimally control traffic lights through a process of experimentation and exploration. The first agent in this set makes use of the Soft Actor Critic (SAC) RL algorithm whereas the second agent makes use of the Proximal Policy Optimisation (PPO) algorithm. The second set of intelligent traffic light controllers make use of Imitation Learning (IL) algorithms which attempt to learn how to optimally control traffic light by inferring the reward function used by experts in the demonstration. This set of controllers consists of three agents. The first agent makes use of Behavioural Cloning (BC), the second makes use Generative Adversarial Imitation Learning (GAIL) and the third agent is also a GAIL agent which is first pre-trained using Behavioural Cloning (BC). Each of these five agents are then trained to optimise the management of the traffic lights in each of the three scenarios in our traffic simulator. Moreover, we also develop a baseline traffic light controller that makes use of a fixed-time round robin strategy for changing lights, where a light in the junction stays in green for exactly the minimum allowed time of five seconds before moving on to an intermediary amber for two seconds and finally red as the next light turns green. For each of the agents in our networks we record the performance obtained in terms of reward that the agent obtains as well as the average queue length and the average trip duration obtained throughout an episode. The same measurements are also recorded for the baseline agent for the purposes of comparison. Our results demonstrate that in scenarios where the baseline controller struggles to minimise congestion and maximise traffic flow, our machine learning agents are able to achieve an improvement in traffic conditions of up to 66%.