Elsevier

Vehicular Communications

Volume 26, December 2020, 100266
Vehicular Communications

Deep reinforcement learning approach for autonomous vehicle systems for maintaining security and safety using LSTM-GAN

https://doi.org/10.1016/j.vehcom.2020.100266Get rights and content

Abstract

The success of autonomous vehicles (AVhs) depends upon the effectiveness of sensors being used and the accuracy of communication links and technologies being employed. But these sensors and communication links have great security and safety concerns as they can be attacked by an adversary to take the control of an autonomous vehicle by influencing their data. Especially during the state estimation process for monitoring of autonomous vehicles' dynamics system, these concerns require immediate and effective solution. In this paper we present a new adversarial deep reinforcement learning algorithm (NDRL) that can be used to maximize the robustness of autonomous vehicle dynamics in the presence of these attacks. In this approach the adversary tries to insert defective data to the autonomous vehicle's sensor readings so that it can disrupt the safe and optimal distance between the autonomous vehicles traveling on the road. The attacker tries to make sure that there is no more safe and optimal distance between the autonomous vehicles, thus it may lead to the road accidents. Further attacker can also add fake data in such a way that it leads to reduced traffic flow on the road. On the other hand, autonomous vehicle will try to defend itself from these types of attacks by maintaining the safe and optimal distance i.e. by minimizing the deviation so that adversary does not succeed in its mission. This attacker-autonomous vehicle action reaction can be studied through the game theory formulation with incorporating the deep learning tools. Each autonomous vehicle will use Long-Short-Term-Memory (LSTM)-Generative Adversarial Network (GAN) models to find out the anticipated distance variation resulting from its actions and input this to the new deep reinforcement learning algorithm (NDRL) which attempts to reduce the variation in distance. Whereas attacker also chooses deep reinforcement learning algorithm (NDRL) and wants to maximize the distance variation between the autonomous vehicles.

Introduction

5G communication provides an inherent support for vehicle-to-everything (V2X) communication. It will bring new capabilities for autonomous vehicles as well as for modern Intelligent Transportation System (ITS) [1], [2]. The feature like very high throughput (i.e. up to multi-Gpbs) with additional uniformity incorporating the wider bandwidths equipped with advance antenna systems would provide an inherent support for various applications especially for autonomous driving. 5G networks can provide 1ms end-to-end latency over a quicker much malleable frame structure; with added functionalities like new up-link RSMA non-orthogonal access would mean that it would meet the lower latency criteria required for efficient autonomous driving more easily and efficiently than any of the previous available technologies [3], [4], [5].

For a very realistic autonomous driving scenario any autonomous vehicle should be able to control and process a sizable volume of information of ITS obtained through huge numbers of sensors and 5G-links. Consistency and reliability of such information plays an important role for maintaining safety and security of autonomous vehicle especially this information may be critical to decrease the chances of road accidents and improve the smooth flow of traffic on ITS roads [6], [7]. But this over dependence for autonomous vehicles makes them extremely vulnerable to various security and safety concerns especially in terms of cyber-physical attacks. An adversary might interrupt the autonomous vehicle's data and impact its consistency and reliability by infusing the incorrect information; which can lead to accidents or can have an adverse impact on the flow of traffic [8], [9]. These types of traffic flow interruptions may also influence the interdependent structure connected to ITSs for example 5G-Communication systems or power grids etc. [10], [11].

In recent past various security mechanisms have been suggested for tackling the security concerns present in autonomous vehicles for maintaining the optimum safe intra-vehicle distance [12], [13], [14], [15], [16]. In [12] the key susceptibilities of a vehicle's controller have been presented and several intrusion detection algorithms have been suggested to overcome these susceptibilities. [13] presented the concept of long-range wireless attacks on autonomous vehicle in which existing security mechanism can be interrupted by controller area network which have impact on optimum safe intra-vehicle distance. [14] discussed the security challenges related to the plug-in electric vehicles which have impact on the power system. A detailed study on the security threats and safety systems in the vehicular networks has been introduced in [15]. [16] presents a deep reinforcement learning based solution for maintaining optimum safe intra-vehicle distance.

Moreover, the vehicular network security issues and their related solutions have been also presented in [17], [18], [19], [20], [21]. [17] lists the security and safety susceptibilities present in the cooperative vehicular networks and try to analyze and optimize the computational cost for beacon signal encryption scheme through using the concept of short-term authentication process. In [18] the authors focused on data infusion attacks and presented the solution to overcome these security issues through the concept of multi-source filters. [19] suggested the idea of merging angle of arrival and Doppler effect to improve security aspect for beacon signals. Furthermore, in [20] solution to spoofing and denial of service attacks through use of collaborative control mechanism have been proposed. A detailed review on vehicular networks and their interaction with modern ITS have been exhibited in [21].

The key problem with these above-mentioned security solutions is that they do not consider the interdependence present between the cyber domain and physical domain of autonomous vehicle while formulating the security and safety resolution. Additionally, there is no proper work in modeling the adversary actions and objectives. In-fact by taking into account the cyber-physical inter-dependencies of adversary's acts and objectives will enable us to provide the better security and safety mechanism. The only work that focus on these inter-dependencies is [16] but it lacks in terms of performance and effectiveness. Thus, an effective safety and security mechanism for autonomous vehicles on modern ITS should be flexible and robust to these attacks on intra-vehicle sensors and for vehicle to vehicle communication. Besides this, all the current works on ITS security usually believe a stable state for the adversary's actions, but in-fact in reality this might not be the case as the adversary may alter its approach adaptively to cause a severe effect of its acts.

The major contribution of this paper is to present a new Deep reinforcement learning algorithm (NDRL) structure that will provide secured autonomous vehicle (AVh) control dynamic system. It is based upon car following model [22], [23]. In this approach focus is on the control system of autonomous vehicle that uses leading autonomous vehicle to adjust its speed and maintain safe-optimal distance. This type of infrastructure requires considering two very important key features:

  • a.

    Autonomous vehicle's Sensors Data.

  • b.

    Autonomous vehicle's Beaconing Signal.

In this work we will consider four important intra-vehicle resources that can point towards an effective safe-optimal distance i.e.

  • 1.

    Intra-vehicle Camera Sensor:

    Optical Information from the camera can be used for the measurement of the speed of moving objects around the autonomous vehicle. Mainly there are two approaches being used for using camera as speed estimator i.e. Inter-frame flow measurement (Optical-method) and Intra-frame measurement (motion-blur-method). Moreover, Camera sensor is also being used for other type of applications as well like number plate recognition system. In this work we have considered the Camera senor using the properties of [24].

  • 2.

    Intra-vehicle Radar Sensor:

    Conventionally speed measurement systems in many vehicles uses RADAR technology. Although RADAR method of estimation of speed is more expensive than Camera sensor but it is more traditional and older method with higher accuracy. RADAR uses radio waves which are repeatedly sent and their returns are measured to determine the distance and velocity of next autonomous vehicle. This concept is known as Doppler effect.

  • 3.

    Smart Roadside Units (sRSUs):

    Smart Roadside units are now being considered for estimating speed of autonomous vehicle. As sRSUs in 5G-V2X have real time connectivity with the autonomous vehicles on the road. And these autonomous vehicles will keep updating their speed and other road condition reports to sRSUs. Thus, sRSUs can be very helpful in maintaining a safe optimal distance between the two autonomous vehicles as it will update the following vehicle about the speed of its leading vehicle.

  • 4.

    Inter-vehicle Beaconing:

    Another important technology that can be used for relying key information (speed, distance etc.) between the two autonomous vehicles is beacon signal between the vehicles i.e. V2V communication for this purpose the Dedicated Short Range Communication (DSRC) (IEEE 802.11p)-based wireless communication technology is used that enables highly secure, high-speed direct communication between the autonomous vehicles. Thus, key information like speed and distance between the two autonomous vehicles can be exchanged.

We have assumed that the adversary can infuse fault data to concede the system and the autonomous vehicle's major objective is to optimally manage its velocity while the attacker is injecting faulty data. To study the relation between autonomous vehicle and adversary, we will formulate this challenge as a non-cooperative game theory and examine its Nash equilibrium. Though finding this Nash equilibrium will be difficult to the continuous action sets that consists of autonomous vehicle's velocity and distance spacing. For overcoming this issue, we are presenting the concept of deep neural network (DNNs) based upon long-short-term-memory (LSTM)-Generative adversarial network (GAN) units. The adversary will try to obtain the previous autonomous vehicle's dynamics information and input it to a new deep reinforcement learning algorithm (NDRL). At the same time the autonomous vehicle's will use NDRL algorithm to discover the best possible assessment from its closer autonomous vehicle's velocity by linking the sensor data readings.

Therefore, the NDRL algorithm for the adversary aims to mislead the autonomous vehicle and introduce deviation between the autonomous vehicles i.e. impacting optimal safe distance spacing while autonomous vehicle tries to minimize this deviation.

Section snippets

System model

First of all we introduce the concept of smart Intelligent Transportation System (ITS) comprising of smart roads which includes autonomous vehicles (AVhs), Smart Roadside Units (sRSUs) using 5G communication links etc. Every AVhj is considered to be equipped with a camera that can be used to take images from its surroundings, smart radar scanner for calculating the distance between the objects present around AVhj and transceiver for transmitting important acceleration speed position (AVP)

Performance evaluation

In this section we will evaluate the proposed work and compare its performance with respect to the previous suggested works. The proposed algorithm is simulated in MATLAB i.e. All the simulation on the vehicle dynamics, communication graph, and the sensor attack on vehicles are implemented in MATLAB.

Conclusions

In this paper, we have presented a new adversarial Deep Reinforcement Learning method (NDRL) that uses Long-Short-Term-Memory (LSTM)-Generative Adversarial Network (GAN) model to provide safety and security mechanism necessary for autonomous vehicles on smart ITS roads equipped with 5G communication links. This method is able to provide effective measure for data infusion attacks that are aim at disrupting the flow of traffic or cause autonomous vehicles to collide leading to accident. To

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (38)

  • A. Sheik et al.

    Key Security Challenges for Cloud-Assisted Connected and Autonomous Vehicles

    (2019)
  • F. Kargl et al.

    Secure vehicular communication systems: implementation, performance, and research challenges

    IEEE Commun. Mag.

    (2008)
  • S.M. Dibaji et al.

    A systems and control perspective of cps security

    Annu. Rev. Control

    (2019)
  • A. Ferdowsi et al.

    A colonel blotto game for interdependence-aware cyber-physical systems security in smart cities

  • P. Kleberger et al.

    Security aspects of the in-vehicle network in the connected car

  • S. Woo et al.

    A practical wireless attack on the connected car and security protocol for in-vehicle can

    IEEE Trans. Intell. Transp. Syst.

    (2014)
  • H. Chaudhry et al.

    Security concerns of a plug-in vehicle

  • I. Studnia et al.

    Survey on security threats and protection mechanisms in embedded automotive networks

  • A. Ferdowsi et al.

    Robust deep reinforcement learning for security and safety in autonomous vehicle systems

  • Cited by (48)

    • Artificial Intelligence techniques to mitigate cyber-attacks within vehicular networks: Survey

      2022, Computers and Electrical Engineering
      Citation Excerpt :

      Alongside, they used a tilt license plate correction algorithm to adapt the practical scene to improve the efficiency of the designed model. For network attack detection and prevention, authors in [69] proposed a DRL approach based on LSTM and generative adversarial network (GAN) to increase the security and the safety of AVs equipped with 5G communication. The model tries to detect attackers attempting to inject erroneous data into the sensor readings of the autonomous vehicle to disrupt the safe and optimal distance between autonomous vehicles.

    • Cyber-security and reinforcement learning — A brief survey

      2022, Engineering Applications of Artificial Intelligence
      Citation Excerpt :

      Fig. 9 summarizes the common topics for existing literature under the IDS, IPS, IoT, and IAM domains. Many topics like computation and data offloading (Huang et al., 2020), anti-jamming (Aref et al., 2017), interference avoidance (Suwannalai and Polprasert, 2020), navigation (Rasheed et al., 2020), and authentication (Xiao et al., 2021) also fall under the category of networks. An important issue that needs to be addressed in these applications is the high generalization ability for the high-variance and dynamic network traffic conditions commonly encountered in all networks.

    View all citing articles on Scopus
    View full text