Elsevier

Computer Networks

Volume 216, 24 October 2022, 109238
Computer Networks

Deep learning for online computation offloading and resource allocation in NOMA

https://doi.org/10.1016/j.comnet.2022.109238Get rights and content

Abstract

The limited battery capacity and low computing capability of wireless Internet of Things (IoT) devices can hardly support computation-intensive and delay-sensitive applications. While recent development of wireless power transfer (WPT) and mobile edge computing (MEC) technologies help IoT devices harvest energy and offload computation tasks to edge servers. While it is still challenging to design an efficient offloading policy to improve the performance of the IoT network. In this article, we consider a MEC network that has WPT capability and adopts the non-orthogonal multiple access (NOMA) technology to offload tasks partially. Our goal is to propose an online algorithm to optimize resource allocation under a wireless dynamic channel scenario. In order to obtain the optimal offloading decision and resource allocation efficiently, we propose a Deep Reinforcement learning-based Online Sample-improving (DROS) framework which implements a deep neural network to input the discretized channel gains to obtain the optimal WPT duration. Based on the WPT duration derived by DNN, we design an optimization algorithm to derive the optimal energy proportion for offloading data. Numerical results verify that compared with traditional optimization algorithms, our proposed DROS has significantly sped up convergence for better solutions.

Introduction

Constrained by aesthetic appearance and production costs, wireless devices (WDs) are limited in battery capacity and computing capability, which leads to a bottleneck in the computing power and the lifetime of WDs. With the development of the smart city, intelligent home automation, and autonomous driving, the Internet of Things (IoT) devices are expected to take responsibility for processing large-scale intensive and delay-sensitive tasks. Hence, how to make the battery of WDs continuously and stably charged over the air, and avoid replacement has become the primary consideration [1]. Fortunately, the advance in wireless power transfer (WPT) technology makes this problem easier [2], [3]. Meanwhile, the mobile edge computing (MEC) technology [4], [5], which can offload tasks from wireless devices to the MEC server in a low-latency manner, is proposed to reduce computing delay and energy consumption [6], [7] for the IoT devices.

The wireless power supply MEC network, which combines the WPT and MEC technology, is considered as a feasible and promising approach to address the energy shortage and low computing power of the traditional IoT networks [8], [9]. In this article, we consider a wireless-powered MEC system where WDs can obtain radio frequency energy from the access point (AP), and offload tasks to the AP for computing afterward. In the process of offloading tasks, we adopt partial offloading, that is, part of the computing data is executed locally on the wireless devices, and the remaining data is offloaded to the AP for computation. Since the data in WDs are to be transmitted to one AP, it is necessary to adopt an efficient multiple-access wireless transmission scheme. Here, we use non-orthogonal multiple access (NOMA) [10], [11] technology, which allows multiple wireless devices to simultaneously offload tasks to the same AP via different frequency channels. Compared with the traditional orthogonal multiple access (OMA), NOMA technology uses Successive Interference Cancellation (SIC) to reduce the subsequent co-channel interference, thereby avoiding the orthogonal separation of radio resources. Therefore, NOMA technology has attracted the attention of many scholars, especially in the fields of the efficiency of radio resource utilization and the improvement of system throughput [12].

The wireless-powered MEC system makes it feasible for low-power WDs to communicate and compute battery-free. However, this integrated system is still facing many challenges. Specifically, on one hand, the wireless channel changes frequently in reality [13], so it is necessary to design an efficient algorithm with a short processing time to improve the network performance. Conventional MEC computing paradigms without WPT technology focus on solving resource allocation problems by using a unique scheme [14], [15], [16]. A wireless-powered network usually has many constraints because of the practical limitations. Hence, how to improve the performance of edge computing under a complex scenario attracts much scholars’ attention [17]. On the other hand, it is hard to jointly optimize resource allocation and task offloading problems. Some works design an offloading algorithm to maximize the computation rate. [18] adopted a decoupling optimization algorithm to optimize binary offloading mode selection, then employed the coordinate descent method to settle alternating direction multiplier decomposition so as to maximize the weighted sum of computation rate.

However, most of the available researches which adopt the traditional optimization method always need multi-iterations. The iteration-based optimization algorithm may result in an intolerable processing delay which cannot satisfy the time-varying channel in the real scenario. Compared with the previous research works, our work explores a DRL-based online offloading (DROS) framework to quickly solve a partial offloading problem so as to maximize the computation rate in the wireless-powered MEC system.

In this paper, we consider a wireless-powered MEC network in which the tasks are arbitrarily assigned and executed by different users. It consists of one AP and multiple WDs as shown in Fig. 1. Our goal is to jointly optimize the WPT duration and energy allocation in a time-varying wireless channel environment. However, this problem is non-convex and hard to settle. To tackle this issue, we design a Deep Reinforcement learning-based Online Sample-improving (DROS) framework to maximize the sum of computation rates, that is, the number of processed bits during a unit of time. Our contributions are mainly as follows:

  • 1.

    The proposed DROS framework can learn optimal WPT duration from the past improving experience and solve the energy allocation problem by optimization. It can achieve a near-optimal solution by using a rather shorter processing time compared with the optimization method.

  • 2.

    We design an additional sampling exploring policy to utilize the exploration nature of deep learning in order to speed up convergence efficiency and improve the outputted policy.

  • 3.

    Numerous results show that the proposed algorithm has a significantly better performance than benchmark schemes.

The remainder of this paper is organized as follows. Section 2 presents the review of related works in the literature. In Section 3, the system model and problem formulation are illustrated. Furthermore, Section 4 investigates the designs of the DROS framework. Simulation results are presented in Section 5. Finally, we conclude this work in Section 6.

Section snippets

Related work

The research issues in MEC have attracted much attention in recent years [4], [19], [20]. Paper [4] considered the service offloading in the MEC. The authors design an approximate algorithm to decide the edge servers for offloading different services. Compared with service offloading, task offloading usually considers a single service scenario and focuses on improving network performance by optimizing bandwidth, transmit power, and so on. In [21], the author proposed an efficient offloading

System model

A wireless powered MEC network which includes an AP and K fixed WDs (illustrated as a set K{1,2,k...,K}) is shown in Fig. 1, and every device is equipped with a single antenna. Each WD contains a rechargeable battery for the normal operation and harvesting of energy. We assume that the AP owns much higher computing capability than the WDs, so that each WD may compute part of the data locally and transfer the rest of the data to the AP [38]. In this article, we adopt NOMA technology as the

The DROS algorithm

In this part, we propose a DRL-based algorithm DROS, which is illustrated in Fig. 2. Generally, a DRL-based DNN leads to learning WPT duration at. We quantize at into M candidate WPT durations and choose the best at which can achieve the maximal computation rate with the optimal energy allocation. The acquired at and corresponding ht are added to the memory to update the DNN.

Numerical results

In this section, we implement our DROS algorithm in python-3.6 with PyTorch-0.4.0. All the simulations are carried out on a desktop with an Intel Core i5-4200 3 GHz CPU and 8 GB memory. The experimental parameters are set as follows: the parameters of P=3 Watts from Powercast TX91501-3 W are used for the energy transmitter at the AP, while the energy collection efficiency μ is set as 0.7 according to the energy receiver P2110 Powerharvester, which is equipped at each WD. In addition, according

Conclusion and future work

In this paper, we propose an online algorithm DROS for the time-varying channel scenario based on NOMA technology and maximize the computing rate in wireless MEC networks by partial offloading. The distinction of our DROS algorithm can be listed as follows. Firstly, the WPT duration from AP to WDs is improved through DNN, moreover, the duration generated by DNN is improved by the previous learning experience. Secondly, we accelerate the convergence of the algorithm by using an additional

CRediT authorship contribution statement

Juncui Niu: Conceptualization, Methodology, Software, Investigation, Formal analysis, Writing – original draft. Shubin Zhang: Data curation, Conceptualization, Funding acquisition, Resources, Supervision, Writing – original draft, Writing – review & editing. Kaikai Chi: Visualization, Investigation. Guanqun Shen: Resources, Supervision. Wei Gao: Software, Validation.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 61872322 and Grant 62072410 and in part by the Zhejiang Provincial Natural Science Foundation of China under Grant LR20F020003 and Grant LQ22F020009.

Juncui Niu, received the B.E. degree in computer science from Hebei Normal University , Hebei, China, in 2008, where she received the M.E. degree with the School of Zhejiang University, in 2018. Currently, she is pursuing the Ph.D. degree at Zhejiang University of Technology, Hangzhou, China. Her research interests include wireless powered communication networks and energy-harvesting cognitive radio networks.

References (44)

  • BiS. et al.

    Wireless powered communication: Opportunities and challenges

    IEEE Commun. Mag.

    (2015)
  • ChiK. et al.

    Energy provision minimization in wireless powered communication networks with network throughput demand: TDMA or NOMA?

    IEEE Trans. Commun.

    (2019)
  • ZhangS. et al.

    Energy management for secure transmission in wireless powered communication networks

    IEEE Internet Things J.

    (2022)
  • PoularakisK. et al.

    Service placement and request routing in MEC networks with storage, computation, and communication constraints

    IEEE ACM Trans. Netw.

    (2020)
  • GuoK. et al.

    Online learning based computation offloading in MEC systems with communication and computation dynamics

    IEEE Trans. Commun.

    (2020)
  • YiC. et al.

    A multi-user mobile computation offloading and transmission scheduling mechanism for delay-sensitive applications

    IEEE Trans. Mob. Comput.

    (2019)
  • GuoH. et al.

    UAV-enhanced intelligent offloading for internet of things at the edge

    IEEE Trans. Ind. Inf.

    (2019)
  • LiuJ. et al.

    Smart and resilient EV charging in SDN-enhanced vehicular edge computing networks

    IEEE J. Sel. Areas Commun.

    (2020)
  • E. Mustafa, J. Shuja, A.I. Jehangiri, S. Din, F. Rehman, S. Mustafa, T. Maqsood, A.N. Khan, Joint wireless power...
  • DaiL. et al.

    A survey of non-orthogonal multiple access for 5G

    IEEE Commun. Surv. Tutor.

    (2018)
  • ZhangH. et al.

    Energy efficient resource allocation in terahertz downlink NOMA systems

    IEEE Trans. Wirel. Commun.

    (2020)
  • WangK. et al.

    Sub-channel scheduling, task assignment, and power allocation for OMA-based and NOMA-based MEC systems

    IEEE Trans. Commun.

    (2020)
  • WangY. et al.

    Adaptive power and rate control for real-time status updating over fading channels

    IEEE Trans. Wirel. Commun.

    (2021)
  • PanY. et al.

    Cost minimization for cooperative computation framework in MEC networks

    IEEE Trans. Wirel. Commun.

    (2021)
  • LiL. et al.

    Resource allocation for NOMA-MEC systems in ultra-dense networks: A learning aided mean-field game approach

    IEEE Trans. Wirel. Commun.

    (2020)
  • ZhuJ. et al.

    Resource allocation for hybrid NOMA MEC offloading

    IEEE Trans. Wirel. Commun.

    (2020)
  • XiaS. et al.

    Online distributed offloading and computing resource management with energy harvesting for heterogeneous MEC-enabled IoT

    IEEE Trans. Wirel. Commun.

    (2020)
  • BiS. et al.

    Computation rate maximization for wireless powered mobile-edge computing with binary computation offloading

    IEEE Trans. Wirel. Commun.

    (2018)
  • TanL.T. et al.

    Mobility-aware edge caching and computing in vehicle networks: A deep reinforcement learning

    IEEE Trans. Veh. Technol.

    (2018)
  • ZhouF. et al.

    Computation rate maximization in UAV-enabled wireless-powered mobile-edge computing systems

    IEEE J. Sel. Areas Commun.

    (2018)
  • AlameddineH.A. et al.

    Dynamic task offloading and scheduling for low-latency IoT services in multi-access edge computing

    IEEE J. Sel. Areas Commun.

    (2019)
  • MaoY. et al.

    Stochastic joint radio and computational resource management for multi-user mobile-edge computing systems

    IEEE Trans. Wirel. Commun.

    (2017)
  • Cited by (8)

    View all citing articles on Scopus

    Juncui Niu, received the B.E. degree in computer science from Hebei Normal University , Hebei, China, in 2008, where she received the M.E. degree with the School of Zhejiang University, in 2018. Currently, she is pursuing the Ph.D. degree at Zhejiang University of Technology, Hangzhou, China. Her research interests include wireless powered communication networks and energy-harvesting cognitive radio networks.

    Shubin Zhang, received the B.S. degree in computer science from Xidian University, Xi’an, China, in 2014, where he received the Ph.D. degree with the School of Cyber Engineering, in 2020. He is currently a Lecturer in the School of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China. His research interests include device-to-device communications, edge computing and IoT networks.

    Kaikai Chi received the B.S. and M.S. degrees from Xidian University, Xi’an, China, in 2002 and 2005, respectively, and the Ph.D. degree from Tohoku University, Sendai, Japan, in 2009. He is currently a professor in the School of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China. His current research focuses on wireless cellular network, wireless ad hoc network and wireless sensor network. He was the recipient of the Best Paper Award at the IEEE Wireless Communications and Networking Conference in 2008. He has published more than 40 referred technical papers in proceedings and journals like IEEE Transactions on Wireless Communications, IEEE Transactions on Vehicular Technology, IEEE Transactions on Parallel and Distributed Systems, etc.

    Guanqun Shen received the B.A. degree in computer science from Zhejiang University City College, Hangzhou, China, in 2011, and the MSc degree in mathematical finance from University of York, UK, in 2013. He is currently pursuing his PhD in the School of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China. His research interests include device-to-device communications, edge computing and IoT networks.

    Wei Gao received the B.E. degree in communication engineering and the Ph.D. degree of information and communication engineering from the Huazhong University of Science and Technology (HUST) in 2014 and 2020, respectively. He is currently an Engineer with China Electric Power Research Institute and the School of Electronic Information and Communications, HUST. His research interests include wireless network access, radio resources allocation, and massive MIMO.

    View full text