Distributed computation offloading method based on deep reinforcement learning in ICV☆
Introduction
With the rapid development of Intelligent Connected Vehicles (ICVs), more and more intelligent applications appear, such as autonomous driving, video-aided real-time navigation, and interactive gaming [1], [2]. However, the computing resources of the vehicle are not sufficient to support the local processing of delay sensitive applications within a short time. To cope with this issue, computation task offloading is a pretty good solution to decrease the execution delays [3], [4], where tasks are pushed to the adjacent servers, such as Multi-Access Edge Computing (MEC) by wireless channel. As shown in Fig. 1, the vehicle offloads the computing task to the Road Side Unit (RSU). It can provide much more abundant computing and storage resources than vehicles, so the execution time is less. However, the communication between vehicle and infrastructure or RSU may cause delay sensitive services to fail. And using infrastructure or RSU to process and store the collected information is expensive. At the same time, if all End Devices (ED) want to offload tasks to MEC, this may bring a great burden to the CPU of MEC [5], resulting in the lack of MEC resources. We can also find that if most vehicles unload the tasks to the edge server to obtain a shorter execution time, the on-board resources will not be fully utilized. In order to not be limited by MEC resources and make full use of the on-board computing resources in Internet of Vehicles (IoV) [6], we need a more effective allocation scheme of vehicular resources [7], as shown in Fig. 1, the vehicle offloads the computing task to the surrounding vehicles. Due to the development of hardware and software, vehicles are more powerful than before, which will possess more and more abundant resources in the future, such as on-board storage and computing resources. Vehicles can be considered as ”moving computers”. Making full use of the vehicle resources in IoV can provide users with flexible, on-demand and nearby services [8], reduce the processing delay of tasks, and enhance the scalability of device computing services. At the same time, the development and utilization of vehicle resources can supplement the computing services of MEC, improve the utilization rate of computing resources, meet the burgeoning demand of large-scale computing intensive unloading tasks [9] and greatly improve the service experience of users.
In this paper, we assume that many Moving Vehicles (MVs) are combined into a Resource Pool (RP), and there are profuse idle resources for tasks execution. To make full use of resources in IoV, we need to research the effective utilization of RP to decrease execution time.
However, there exist some challenges before we leverage RP to improve the execution time of a multifaceted task without MEC services. The challenges are summarized as following:
- •
The mobility of vehicles. Resulting from the high-speed movement of MVs, the link connectivity between vehicles in IoV is unstable [10], [11]. If we allocate computation tasks to the surrounding vehicles, we need to know how much service time the surrounding MVs can provide for the Requested Vehicle (RV), so that results of offloading tasks can return to RV in time. The service time provided by surrounding MVs directly determines the overall performance of the task execution.
- •
Limited vehicular resources for a single vehicle. The resource pool is comprised of numerous moving vehicles, but the resource of each vehicle is not enough to calculate a complex task in the time required. Splitting a complex task into many small tasks is an effectual way to reduce the execution time of each task. And assigning these tasks to surrounding vehicles for execution is a promising way to utilize vehicular resources properly [12].
- •
How to assign these sub-tasks. In this paper, we split a complex task into many tiny pieces randomly. We need to allocate these small tasks with different sizes to the surrounding vehicles, which have diverse computing resources, so the optimization of the allocation scheme of these small tasks is a Non-deterministic Polynomial hard (NP-hard) problem.
It is a promising remedy for vehicles to offload tasks into MVs, which can make full use of every vehicle’s idle computing resources, and run intelligent applications well in scenarios such as highways with insufficient MEC services. The main contributions of this paper are summarized as following:
- •
Based on the RP formed by vehicles, we develop a task offloading scheme merely relying on Vehicle to Vehicle (V2V) communication. The RV offloads tasks to surround MVs to gain a shorter task execution time.
- •
We build a moving model of vehicles on the highway to calculate the relative distance of them, determining the service time that MVs can provide for RV.
- •
We design an algorithm Distributed Offloading Deep Q-learning Network (DODQN), which is based on Deep Q-learning Network (DQN) [13], to solve the problem of assigning tasks to vehicles, and we compare the results with other schemes. Simulation results verify the effectiveness of the proposed scheme.
The remainder is structured as follows. Section 2 introduces the related work. The system model is established in Section 3. Section 4 analyzes the formula and solution method of the problem and obtains the optimal task allocation. Finally, Section 5 shows the simulation results. Section 6 summarizes this article.
Section snippets
Related work
Due to the computing resources of the vehicle are limited in IoV, computation offloading is the main solution. Only when computing tasks are offloaded effectively, can we obtain efficient and low-latency services. However, computation offloading will bring new problems, such as the allocation of computing resources, the increase in communication cost, how to offload computation tasks, how to determine the offload routing, and how many computing tasks can be offloaded [14].
In order to meet the
System model
This section describes the system architecture for distributed offloading in Vehicular Edge Computing (VEC). As shown in Fig. 2, we present the system model of VEC on the highway. We design a DRL-based mechanism to employ several surrounding MVs for task execution. For service provision in VEC, the requests from MVs on the road can be realized and processed directly by surrounding MVs. The computational resources of MVs are scheduled under the First Come First Served (FCFS) rule to execute the
Solution
In this section, we transform the Max–Min problem into an optimal offloading problem and establish a Markov Decision Process (MDP) for this problem. In order to solve the MDP, an approach based on DQN is proposed, which combine deep learning with Q-learning to solve the continuous state space problem. The flowchart of DQN is shown in Fig. 7.
Simulation
In this section, we make a numerical simulation to estimate the performance of our proposed algorithm and the validity of splitting task. Simulation parameters are shown in Table 2.
Conclusion
In order to address the problem of insufficient computation services of MEC, this paper proposes a distributed computation offloading strategy in IoV. The genetic algorithm and deep reinforcement learning are utilized to solve the model to obtain the optimal tasks allocation scheme. This offloading strategy can ensure the delay requirements of a big intelligent task in the scenario of insufficient MEC resources, thereby ensuring a good user experience. The simulation results prove that the
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Chen Chen received the B.Eng., M.Sc. and Ph.D. degrees in telecommunication from Xidian University, Xi’an, China, in 2000, 2006, and 2008, respectively. He is currently a Professor with the Department of telecommunication in Xidian University, and a member of the The State Key Laboratory of Integrated Service Networks in Xidian University. He is also the Director of the Xi’an Key Laboratory of Mobile Edge Computing and Security, and the Director of the intelligent transportation research
References (32)
- et al.
A data dissemination scheme based on clustering and probabilistic broadcasting in vanets
Veh. Commun.
(2018) - et al.
Caching in vehicular named data networking: Architecture, schemes and future directions
IEEE Commun. Surv. Tutor.
(2020) - et al.
Cognitive computing and wireless communications on the edge for healthcare service robots
Comput. Commun.
(2020) - et al.
Architecture and performance evaluation of distributed computation offloading in edge computing
Simul. Model. Pract. Theory
(2020) - et al.
Distributed cooperative computation offloading in multi-access edge computing fiber–wireless networks
Opt. Commun.
(2019) - et al.
Parked vehicular computing for energy-efficient internet of vehicles: A contract theoretic approach
IEEE Internet Things J.
(2018) - et al.
Green resource allocation based on deep reinforcement learning in content-centric iot
IEEE Trans. Emerg. Top. Comput.
(2018) - et al.
Fog vehicular computing : augmentation of fog computing using vehicular cloud computing
IEEE Veh. Technol. Mag.
(2017) - et al.
Scalable fog computing with service offloading in bus networks
- et al.
Toward offloading internet of vehicles applications in 5g networks
IEEE Trans. Intell. Transp. Syst.
(2020)
Mobile-edge computing for vehicular networks: A promising network paradigm with predictive off-loading
IEEE Veh. Technol. Mag.
Future communications and energy management in the internet of vehicles: Toward intelligent energy-harvesting
IEEE Wirel. Commun.
Distributed vehicular computing at the dawn of 5g: a survey
Vehicular fog computing: A viewpoint of vehicles as the infrastructures
IEEE Trans. Veh. Technol.
Joint node selection and resource allocation for task offloading in scalable vehicle-assisted multi-access edge computing
Symmetry
A connectivity-prediction-based dynamic clustering model for vanet in an urban scene
IEEE Internet Things J.
Cited by (83)
A relay-assisted parallel offloading strategy for multi-source tasks in internet of vehicles
2024, Sustainable Energy Technologies and AssessmentsA comprehensive survey on reinforcement-learning-based computation offloading techniques in Edge Computing Systems
2023, Journal of Network and Computer ApplicationsA Multiagent Meta-Based Task Offloading Strategy for Mobile-Edge Computing
2024, IEEE Transactions on Cognitive and Developmental SystemsDistributed Task Offloading Method Based on Federated Reinforcement Learning in Vehicular Networks with Incomplete Information
2024, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)Distributed Task Offloading for IoAV Using DDP-DQN
2024, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Chen Chen received the B.Eng., M.Sc. and Ph.D. degrees in telecommunication from Xidian University, Xi’an, China, in 2000, 2006, and 2008, respectively. He is currently a Professor with the Department of telecommunication in Xidian University, and a member of the The State Key Laboratory of Integrated Service Networks in Xidian University. He is also the Director of the Xi’an Key Laboratory of Mobile Edge Computing and Security, and the Director of the intelligent transportation research laboratory in Xidian University. He was a visiting professor at the department of EECS in the University of Tennessee and the department of CS in the University of California. He serves as General Chair, PC Chair, Workshop Chair or TPC Member of a number of conferences. He has authored/co-authored 2 books, over 130 scientific papers in international journals and conference proceedings. He has contributed to the development of 5 copyrighted software systems and invented over 100 patents. He is also a senior member of China Computer Federation (CCF) and China Institute of Communications (CIC).
Yuru Zhang received the B.Eng. degree in communication engineering from Lanzhou Jiaotong University, Lanzhou, China, in 2019. She is currently pursuing the master’s degree with Xidian University. Her major is transportation information engineering and control. Her research interests include Internet of Vehicle, wireless communication and Space-Air-Ground Integrated Network.
Zheng Wang received the B.Eng. in communication engineering from Xidian University, Xi’an, China, in 2017. Since 2017, he has been working on Master’s Degree in Xidian University, Xi’an, China. His research interests include mobile edge computing, computer engineering, traffic information and control engineering.
Shaohua Wan received the joint Ph.D. degree from the School of Computer, Wuhan University and the Department of Electrical Engineering and Computer Science, Northwestern University, USA in 2010. Since 2015, he has been holding a post-doctoral position at the State Key Laboratory of Digital Manufacturing Equipment and Technology, Huazhong University of Science and Technology. From 2016 to 2017, he was a visiting professor at Department of Electrical and Computer Engineering, Technical University of Munich, Germany. He is currently an associate professor at School of Information and Safety Engineering, Zhongnan University of Economics and Law. His main research interests include deep learning for Internet of Things and edge computing. He is an author of over 90 peer-reviewed research papers and books. He is a senior member of IEEE.
Qingqi Pei received the B.Eng., M.Eng. and Ph.D. degrees in Computer Science and Cryptography from Xidian University, in 1998, 2005 and 2008, respectively. He is now a Professor and member of the State Key Laboratory of Integrated Services Networks, also a Professional Member of ACM and Senior Member of IEEE, Chinese Institute of Electronics and China Computer Federation. He is also the director of the Shaanxi Key Laboratory of Blockchain and Secure Computing. His research interests focus on digital contents protection and wireless networks and security.
- ☆
This work was supported by the National Key Research and Development Program of China (2019YFE0196600), the National Natural Science Foundation of China (62072360, 61902292, 62001357, 62072359, 62072355), the key research and development plan of Shaanxi province (2019ZDLGY13-07, 2019ZDLGY13-04, 2020JQ-844), the key laboratory of embedded system and service computing (Tongji University, China) (ESSCKF2019-05), Ministry of Education, the Xi’an Science and Technology Plan, China (20RGZN0005) and the Xi’an Key Laboratory of Mobile Edge Computing and Security, China (201805052-ZD3CG36).