Traffic flow control using multi-agent reinforcement learning

Abstract

One of the technologies based on information technology used today is the VANET network used for inter-road communication. Today, many developed countries use this technology to optimize travel times, queue lengths, number of vehicle stops, and overall traffic network efficiency. In this research, we investigate the critical and necessary factors to increase the quality of VANET networks. This paper focuses on increasing the quality of service using multi-agent learning methods. The innovation of this study is using artificial intelligence to improve the network’s quality of service, which uses a mechanism and algorithm to find the optimal behavior of agents in the VANET. The result indicates that the proposed method is more optimal in the evaluation criteria of packet delivery ratio (PDR), transaction success rate, phase duration, and queue length than the previous ones. According to the evaluation criteria, TSR 6.342%, PDR 9.105%, QL 7.143%, and PD 6.783% are more efficient than previous works.

Introduction

In recent years, existing methods in computer science and especially artificial intelligence have attracted the attention of experts to control traffic. Congestion (density) is a common phenomenon at intersections and long queues of cars behind traffic lights that is seen every day in large cities. This congestion that occurs continuously at different levels and aspects imposes huge costs on communities. Congestion caused by sudden events affecting traffic are accidents, bad weather, technical defects and so on. Intelligent transportation systems are the ones that use telecommunication and control technologies, hardware and software in land transportation systems.

One of the advantages of artificial intelligence and especially reinforcing learning is that they do not need the environment model and the intervention of the designer (human). Meanwhile, multi-way systems have gained a lot of fans in traffic control due to their closeness to the nature of traffic problems. In other words, the learning agent continuously acquires the necessary knowledge for optimal performance through interaction with the environment. The learning agent performs actions in the environment sequentially and receives feedback on those actions in the form of rewards. Due to the uncertainty in traffic environments, their high complexity and dynamics, it is very difficult to gain a primary knowledge of the environment and put it in the mind of the agent. Therefore, it is vital to have a learning mechanism through which the agent can acquire the necessary knowledge while making decisions and intelligently interacting with the uncertain environment.

In this study, we use artificial intelligence and reinforcement learning to create data and datasets that actually include test data, training data, and simulated data. Due to the nature of the traffic problem that is the unavailability of training data, the unavailability of educational data, supervised learning methods are not appropriate to achieve this goal. We need a method that can acquire the necessary knowledge without using training data (unsupervised learning). The use of artificial intelligence and especially reinforced learning as one of the successful approaches in machine learning can provide such an opportunity. In this method, the agent achieves the optimal policy (proper timing of the traffic light) through interacting with the uncertain environment. The learning agent performs actions in the environment sequentially and receives feedback on those actions in the form of rewards.

In traffic lights control, scheduling parameters according to the existing traffic conditions (for example, the number of cars waiting at the intersection and the delay time) to achieve some goals (e.g., maximum number of vehicles crossing the intersection, reduced travel time, reduced queue length and network efficiency) are optimized. A coordination mechanism to find the optimal behavior of all agents is another goal of this paper. In this research, Q-learning algorithm is used for more comprehensive studies (Nazib and Moh, 2021).

Applying reinforcing learning in traffic control, which is a multi-agent problem, is associated with different challenges. The most important challenge is that learning agents (traffic lights) are learning individually and independently, and eventually find the optimal behavior of the individual that will not necessarily lead to the optimal behavior of all agents. Addressing this challenge in traffic control is one of the main goals of this study.

We also address the multi-intersection traffic control problem using artificial intelligence, and in particular reinforcing learning. In this study, we aim to reduce queue length, travel time and increase network efficiency, as well as increase the quality of service of the traffic network.

In this study, we want to achieve high quality of service, low latency and increased response time as evaluation criteria. If the stability of the network is concerned, then the traffic network and all evaluation criteria would be disrupted. It is therefore essential to use prediction methods to be able to manage and control the traffic network. Reinforcement learning as one of the successful approaches in machine learning can provide such an opportunity. To reduce queue length, travel time, increase network efficiency, and increase the quality of service of the traffic network, we have used the Q-Learning algorithm by adjusting its parameters.

Also, by using traffic patterns for highways and maps where there are intersections, we detect intersections with high traffic density and then provide some offers to our users in order to use the network online.

One of the goals we are pursuing in this research is to increase the quality of service in the Vent network. As the quality of service increases, the efficiency of the traffic network increases and our clients are satisfied. In addition, the network can have more bandwidth capacity.

Controlling and managing the traffic network is another goal and if this goal is achieved, we can provide better services and offers to clients. Otherwise, our network is not online and clients cannot access network data. Controlling and managing the traffic network is another goal and if this goal is achieved, we can provide better services and offers to clients. Otherwise, our network is not online and clients cannot access network data.

We assume that, if we use prediction patterns, then we can control and manage traffic. Also, when we use traffic control and artificial intelligence methods, we can control the traffic network and increase the quality of service of the traffic network.

Multiple layers are shared for a series of services in one environment. In the input layer, the massive data of the traffic network is analyzed. In order to control and manage traffic, a series of special programs have been considered to increase user satisfaction. In addition, the network load bar can be also optimized, which requires a series of data centers for very large metrics and scales in order to be able to respond to requests. which requires a series of data centers for very large metrics and scales to be able to respond to requests.

In this layer, a series of phrases are considered to respond to the quality of service, and the problem that arises is related to the level of access. In this layer, calculations are performed flexibly and can be performed by systems (see Fig. 1).

Section 2 provides a review of previous work. Section 3 describes the proposed method. The method used in this study will be explained in Section 3. In Section 5, the traffic-based agent simulation is studied. The design of guidance control based on reinforcement learning (defining environmental states, actions, and rewards) is discussed in Section 3. Simulation results are investigated in Section 5 and finally, Section 5 is devoted to discussion and conclusion.