Distributed computation offloading method based on deep reinforcement learning in ICV

https://doi.org/10.1016/j.asoc.2021.107108Get rights and content

Abstract

With the rapid development of Intelligent Connected Vehicles (ICVs), more effective computation resources optimization schemes in task scheduling are exactly required for large-scale network implementation. We observe that an offloading scheme that almost all tasks are going to be executed in Multi-Access Edge Computing (MEC) servers, which lead to a lot of vehicle resources to be underutilized and put a great burden on severs, is not a good solution for resource utilization. So we first consider the scenario where MEC is not available or enough. We take surrounding vehicles as a Resource Pool (RP). And we propose a distributed computation offloading method to utilize all resources, in which a complex task can be split into many small sub-tasks. How to assign these minor tasks to get a better execution time in RP is a hard problem. The executing time of a complex computing task is a min–max problem. In this paper, a distributed computation offloading strategy based on Deep Q-learning Network (DQN) is proposed to find the best offloading method to minimize the execution time of a compound task. We can demonstrate that the model proposed in this paper can take full advantage of the computing resources of the surrounding vehicles and greatly reduce the execution time of the computation tasks.

Introduction

With the rapid development of Intelligent Connected Vehicles (ICVs), more and more intelligent applications appear, such as autonomous driving, video-aided real-time navigation, and interactive gaming [1], [2]. However, the computing resources of the vehicle are not sufficient to support the local processing of delay sensitive applications within a short time. To cope with this issue, computation task offloading is a pretty good solution to decrease the execution delays [3], [4], where tasks are pushed to the adjacent servers, such as Multi-Access Edge Computing (MEC) by wireless channel. As shown in Fig. 1, the vehicle offloads the computing task to the Road Side Unit (RSU). It can provide much more abundant computing and storage resources than vehicles, so the execution time is less. However, the communication between vehicle and infrastructure or RSU may cause delay sensitive services to fail. And using infrastructure or RSU to process and store the collected information is expensive. At the same time, if all End Devices (ED) want to offload tasks to MEC, this may bring a great burden to the CPU of MEC [5], resulting in the lack of MEC resources. We can also find that if most vehicles unload the tasks to the edge server to obtain a shorter execution time, the on-board resources will not be fully utilized. In order to not be limited by MEC resources and make full use of the on-board computing resources in Internet of Vehicles (IoV) [6], we need a more effective allocation scheme of vehicular resources [7], as shown in Fig. 1, the vehicle offloads the computing task to the surrounding vehicles. Due to the development of hardware and software, vehicles are more powerful than before, which will possess more and more abundant resources in the future, such as on-board storage and computing resources. Vehicles can be considered as ”moving computers”. Making full use of the vehicle resources in IoV can provide users with flexible, on-demand and nearby services [8], reduce the processing delay of tasks, and enhance the scalability of device computing services. At the same time, the development and utilization of vehicle resources can supplement the computing services of MEC, improve the utilization rate of computing resources, meet the burgeoning demand of large-scale computing intensive unloading tasks [9] and greatly improve the service experience of users.

In this paper, we assume that many Moving Vehicles (MVs) are combined into a Resource Pool (RP), and there are profuse idle resources for tasks execution. To make full use of resources in IoV, we need to research the effective utilization of RP to decrease execution time.

However, there exist some challenges before we leverage RP to improve the execution time of a multifaceted task without MEC services. The challenges are summarized as following:

  • The mobility of vehicles. Resulting from the high-speed movement of MVs, the link connectivity between vehicles in IoV is unstable [10], [11]. If we allocate computation tasks to the surrounding vehicles, we need to know how much service time the surrounding MVs can provide for the Requested Vehicle (RV), so that results of offloading tasks can return to RV in time. The service time provided by surrounding MVs directly determines the overall performance of the task execution.

  • Limited vehicular resources for a single vehicle. The resource pool is comprised of numerous moving vehicles, but the resource of each vehicle is not enough to calculate a complex task in the time required. Splitting a complex task into many small tasks is an effectual way to reduce the execution time of each task. And assigning these tasks to surrounding vehicles for execution is a promising way to utilize vehicular resources properly [12].

  • How to assign these sub-tasks. In this paper, we split a complex task into many tiny pieces randomly. We need to allocate these small tasks with different sizes to the surrounding vehicles, which have diverse computing resources, so the optimization of the allocation scheme of these small tasks is a Non-deterministic Polynomial hard (NP-hard) problem.

It is a promising remedy for vehicles to offload tasks into MVs, which can make full use of every vehicle’s idle computing resources, and run intelligent applications well in scenarios such as highways with insufficient MEC services. The main contributions of this paper are summarized as following:

  • Based on the RP formed by vehicles, we develop a task offloading scheme merely relying on Vehicle to Vehicle (V2V) communication. The RV offloads tasks to surround MVs to gain a shorter task execution time.

  • We build a moving model of vehicles on the highway to calculate the relative distance of them, determining the service time that MVs can provide for RV.

  • We design an algorithm Distributed Offloading Deep Q-learning Network (DODQN), which is based on Deep Q-learning Network (DQN) [13], to solve the problem of assigning tasks to vehicles, and we compare the results with other schemes. Simulation results verify the effectiveness of the proposed scheme.

The remainder is structured as follows. Section 2 introduces the related work. The system model is established in Section 3. Section 4 analyzes the formula and solution method of the problem and obtains the optimal task allocation. Finally, Section 5 shows the simulation results. Section 6 summarizes this article.

Section snippets

Related work

Due to the computing resources of the vehicle are limited in IoV, computation offloading is the main solution. Only when computing tasks are offloaded effectively, can we obtain efficient and low-latency services. However, computation offloading will bring new problems, such as the allocation of computing resources, the increase in communication cost, how to offload computation tasks, how to determine the offload routing, and how many computing tasks can be offloaded [14].

In order to meet the

System model

This section describes the system architecture for distributed offloading in Vehicular Edge Computing (VEC). As shown in Fig. 2, we present the system model of VEC on the highway. We design a DRL-based mechanism to employ several surrounding MVs for task execution. For service provision in VEC, the requests from MVs on the road can be realized and processed directly by surrounding MVs. The computational resources of MVs are scheduled under the First Come First Served (FCFS) rule to execute the

Solution

In this section, we transform the Max–Min problem into an optimal offloading problem and establish a Markov Decision Process (MDP) for this problem. In order to solve the MDP, an approach based on DQN is proposed, which combine deep learning with Q-learning to solve the continuous state space problem. The flowchart of DQN is shown in Fig. 7.

Simulation

In this section, we make a numerical simulation to estimate the performance of our proposed algorithm and the validity of splitting task. Simulation parameters are shown in Table 2.

Conclusion

In order to address the problem of insufficient computation services of MEC, this paper proposes a distributed computation offloading strategy in IoV. The genetic algorithm and deep reinforcement learning are utilized to solve the model to obtain the optimal tasks allocation scheme. This offloading strategy can ensure the delay requirements of a big intelligent task in the scenario of insufficient MEC resources, thereby ensuring a good user experience. The simulation results prove that the

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Chen Chen received the B.Eng., M.Sc. and Ph.D. degrees in telecommunication from Xidian University, Xi’an, China, in 2000, 2006, and 2008, respectively. He is currently a Professor with the Department of telecommunication in Xidian University, and a member of the The State Key Laboratory of Integrated Service Networks in Xidian University. He is also the Director of the Xi’an Key Laboratory of Mobile Edge Computing and Security, and the Director of the intelligent transportation research

References (32)

  • ZhangK. et al.

    Mobile-edge computing for vehicular networks: A promising network paradigm with predictive off-loading

    IEEE Veh. Technol. Mag.

    (2017)
  • WangX. et al.

    Future communications and energy management in the internet of vehicles: Toward intelligent energy-harvesting

    IEEE Wirel. Commun.

    (2019)
  • AlhilalA. et al.

    Distributed vehicular computing at the dawn of 5g: a survey

    (2020)
  • HouX. et al.

    Vehicular fog computing: A viewpoint of vehicles as the infrastructures

    IEEE Trans. Veh. Technol.

    (2016)
  • PhamX.-Q. et al.

    Joint node selection and resource allocation for task offloading in scalable vehicle-assisted multi-access edge computing

    Symmetry

    (2019)
  • ChengJ. et al.

    A connectivity-prediction-based dynamic clustering model for vanet in an urban scene

    IEEE Internet Things J.

    (2020)
  • Cited by (83)

    • A Multiagent Meta-Based Task Offloading Strategy for Mobile-Edge Computing

      2024, IEEE Transactions on Cognitive and Developmental Systems
    • Distributed Task Offloading Method Based on Federated Reinforcement Learning in Vehicular Networks with Incomplete Information

      2024, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    • Distributed Task Offloading for IoAV Using DDP-DQN

      2024, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    View all citing articles on Scopus

    Chen Chen received the B.Eng., M.Sc. and Ph.D. degrees in telecommunication from Xidian University, Xi’an, China, in 2000, 2006, and 2008, respectively. He is currently a Professor with the Department of telecommunication in Xidian University, and a member of the The State Key Laboratory of Integrated Service Networks in Xidian University. He is also the Director of the Xi’an Key Laboratory of Mobile Edge Computing and Security, and the Director of the intelligent transportation research laboratory in Xidian University. He was a visiting professor at the department of EECS in the University of Tennessee and the department of CS in the University of California. He serves as General Chair, PC Chair, Workshop Chair or TPC Member of a number of conferences. He has authored/co-authored 2 books, over 130 scientific papers in international journals and conference proceedings. He has contributed to the development of 5 copyrighted software systems and invented over 100 patents. He is also a senior member of China Computer Federation (CCF) and China Institute of Communications (CIC).

    Yuru Zhang received the B.Eng. degree in communication engineering from Lanzhou Jiaotong University, Lanzhou, China, in 2019. She is currently pursuing the master’s degree with Xidian University. Her major is transportation information engineering and control. Her research interests include Internet of Vehicle, wireless communication and Space-Air-Ground Integrated Network.

    Zheng Wang received the B.Eng. in communication engineering from Xidian University, Xi’an, China, in 2017. Since 2017, he has been working on Master’s Degree in Xidian University, Xi’an, China. His research interests include mobile edge computing, computer engineering, traffic information and control engineering.

    Shaohua Wan received the joint Ph.D. degree from the School of Computer, Wuhan University and the Department of Electrical Engineering and Computer Science, Northwestern University, USA in 2010. Since 2015, he has been holding a post-doctoral position at the State Key Laboratory of Digital Manufacturing Equipment and Technology, Huazhong University of Science and Technology. From 2016 to 2017, he was a visiting professor at Department of Electrical and Computer Engineering, Technical University of Munich, Germany. He is currently an associate professor at School of Information and Safety Engineering, Zhongnan University of Economics and Law. His main research interests include deep learning for Internet of Things and edge computing. He is an author of over 90 peer-reviewed research papers and books. He is a senior member of IEEE.

    Qingqi Pei received the B.Eng., M.Eng. and Ph.D. degrees in Computer Science and Cryptography from Xidian University, in 1998, 2005 and 2008, respectively. He is now a Professor and member of the State Key Laboratory of Integrated Services Networks, also a Professional Member of ACM and Senior Member of IEEE, Chinese Institute of Electronics and China Computer Federation. He is also the director of the Shaanxi Key Laboratory of Blockchain and Secure Computing. His research interests focus on digital contents protection and wireless networks and security.

    This work was supported by the National Key Research and Development Program of China (2019YFE0196600), the National Natural Science Foundation of China (62072360, 61902292, 62001357, 62072359, 62072355), the key research and development plan of Shaanxi province (2019ZDLGY13-07, 2019ZDLGY13-04, 2020JQ-844), the key laboratory of embedded system and service computing (Tongji University, China) (ESSCKF2019-05), Ministry of Education, the Xi’an Science and Technology Plan, China (20RGZN0005) and the Xi’an Key Laboratory of Mobile Edge Computing and Security, China (201805052-ZD3CG36).

    View full text