Skip to main content

Advertisement

SpringerLink
Log in

Deep reinforcement learning-based computation offloading and resource allocation in security-aware mobile edge computing

  • Original Paper
  • Published:
Wireless Networks Aims and scope Submit manuscript

Abstract

Owing to the insufficient processing ability of wireless devices (WDs), it is difficult for WDs to process these data within the deadline associated with the quality of service requirements. Offloading computation tasks (workloads) to emerging mobile edge computing servers with small or macro base stations is an effective and feasible solution. However, the offloaded data will be fully exposed and vulnerable to security threats. In this paper, we introduce a wireless communication and computation model of partial computation offloading and resource allocation considering the time-varying channel state, the bandwidth constraint, the stochastic arrival of workloads, and privacy preservation. To simultaneously optimize the computation and execution delays, the power consumption, and the bandwidth resources, we model the optimization problem as a Markov decision process (MDP) to minimize the weighted sum cost of the system. Owing to the difficult problems of lack of priori knowledge and the curse of dimensionality, we propose a decentralized optimization scheme on partial computation offloading and resource allocation based on deep reinforcement learning (DOCRRL). According to the time-varying channel state, the arrival rate of computation workloads, and the signal-to-interference-plus-noise ratio, the DOCRRL algorithm can learn the optimal policy for decision-making under stringent latency and risk constraints that prevent the curse of dimensionality from arising owing to the high-dimensional action space and state space. The numerical results reveal that DOCRRL can explore and learn the optimal decision-making policy without priori knowledge; it outperforms four baseline schemes in simulation environments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. We assume that the computation workloads are a batch images to be processed, such as 1 batch of 10 traffic scene images or face recognition images.

  2. The resources scheduling scheme for MEC servers that can be leveraged to train NNs is beyond the scope of this paper, so it is not explained here.

  3. We assume that 10 face recognition images is a batch of computation workloads and these workloads(1 batch of 10 face recognition images) follows the Poisson distribution with \(\lambda _w\).

References

  1. Kiran, N., Pan, C., Wang, S., & Yin, C. (2020). Joint resource allocation and computation offloading in mobile edge computing for sdn based wireless networks. Journal of Communications and Networks, 22(1), 1–11.

    Article  Google Scholar 

  2. Ahmadabadian, M., Moghaddam, S. K., & Razavizadeh, S. M. (2020). Energy efficiency maximization in fdd massive mimo systems with channel aging. Wireless Networks, 26(2).

  3. Park, S., Kang, Y., Tian, Y., & Kim, J. (2020) Fast and reliable offloading via deep reinforcement learning for mobile edge video computing. In: 2020 International Conference on Information Networking (ICOIN), pp. 10–12

  4. Hadded, L., Charrada, F.B., & Tata, S. (2018). Efficient resource allocation for autonomic service-based applications in the cloud. In: 2018 IEEE International Conference on Autonomic Computing (ICAC), pp. 193–198. IEEE

  5. Du, J., Zhao, L., Feng, J., & Chu, X. (2018). Computation offloading and resource allocation in mixed fog/cloud computing systems with min-max fairness guarantee. IEEE Transactions on Communications, 66(4), 1594–1608.

    Article  Google Scholar 

  6. Liu, Y., Xiong, K., Ni, Q., Fan, P., & Letaief, K. B. (2020). Uav-assisted wireless powered cooperative mobile edge computing: Joint offloading, cpu control, and trajectory optimization. IEEE Internet of Things Journal, 7(4), 2777–2790. https://doi.org/10.1109/JIOT.2019.2958975.

    Article  Google Scholar 

  7. Zhou, F., & Hu, R. Q. (2020). Computation efficiency maximization in wireless-powered mobile edge computing networks. IEEE Transactions on Wireless Communications, 19(5), 3170–3184.

    Article  Google Scholar 

  8. Sun, Y., Xu, L., Tang, Y., & Zhuang, W. (2018). Traffic offloading for online video service in vehicular networks: A cooperative approach. IEEE Transactions on Vehicular Technology, 67(8), 7630–7642.

    Article  Google Scholar 

  9. Pasha, M., & Rahman Khan, K. U. (2018). Scalable and energy efficient task offloading schemes for vehicular cloud computing. International Journal of Computer Networks and Communications (IJCNC), 10,.

  10. Zhan, W., Luo, C., Wang, J., Wang, C., Min, G., Duan, H., & Zhu, Q. (2020). Deep-reinforcement-learning-based offloading scheduling for vehicular edge computing. IEEE Internet of Things Journal, 7(6), 5449–5465.

    Article  Google Scholar 

  11. Sun, F., Hou, F., Cheng, N., Wang, M., Zhou, H., Gui, L., & Shen, X. (2018). Cooperative task scheduling for computation offloading in vehicular cloud. IEEE Transactions on Vehicular Technology, 67(11), 11049–11061.

    Article  Google Scholar 

  12. Liu, J., Kumar, K., Lu, Y.-H.: Tradeoff between energy savings and privacy protection in computation offloading. In: 2010 ACM/IEEE International Symposium on Low-Power Electronics and Design (ISLPED), pp. 213–218 (2010). IEEE

  13. Liu, J., Lu, Y.-H.: Energy savings in privacy-preserving computation offloading with protection by homomorphic encryption. In: Proceedings of the 2010 International Conference on Power Aware Computing and Systems, HotPower, vol. 10, pp. 1–7 (2010)

  14. Goudarzi, M., Zamani, M., & Haghighat, A. T. (2017). A fast hybrid multi-site computation offloading for mobile cloud computing. Journal of Network and Computer Applications, 80, 219–231.

    Article  Google Scholar 

  15. Shahzad, H., & Szymanski, T.H. (2016). A dynamic programming offloading algorithm for mobile cloud computing. In: 2016 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), pp. 1–5. IEEE

  16. Valentino, R., Jung, W.-S., & Ko, Y.-B. (2018). A design and simulation of the opportunistic computation offloading with learning-based prediction for unmanned aerial vehicle (uav) clustering networks. Sensors, 18(11), 3751.

    Article  Google Scholar 

  17. Lyu, X., Tian, H., Jiang, L., Vinel, A., Maharjan, S., Gjessing, S., & Zhang, Y. (2018). Selective offloading in mobile edge computing for the green internet of things. IEEE Network, 32(1), 54–60.

    Article  Google Scholar 

  18. Ning, Z., Dong, P., Kong, X., & Xia, F. (2018). A cooperative partial computation offloading scheme for mobile edge computing enabled internet of things. IEEE Internet of Things Journal

  19. Tang, Q., Xie, R., Yu, F. R., Huang, T., & Liu, Y. (2020). Decentralized computation offloading in iot fog computing system with energy harvesting: A dec-pomdp approach. IEEE Internet of Things Journal, 7(6), 4898–4911. https://doi.org/10.1109/JIOT.2020.2971323.

    Article  Google Scholar 

  20. Wang, C., Liang, C., Yu, F. R., Chen, Q., & Tang, L. (2017). Computation offloading and resource allocation in wireless cellular networks with mobile edge computing. IEEE Transactions on Wireless Communications, 16(8), 4924–4938.

    Article  Google Scholar 

  21. Qian, L. P., Feng, A., Huang, Y., Wu, Y., Ji, B., & Shi, Z. (2018). Optimal sic ordering and computation resource allocation in mec-aware noma nb-iot networks. IEEE Internet of Things Journal, 6(2), 2806–2816.

    Article  Google Scholar 

  22. Tan, Z., Yu, F.R., Li, X., Ji, H., & Leung, V.C. (2017). Virtual resource allocation for heterogeneous services in full duplex-enabled small cell networks with cache and mec. In: 2017 IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), pp. 163–168 . IEEE

  23. Wu, X., Jiang, W., Zhang, Y., & Yu, W. (2019). Online combinatorial based mechanism for mec network resource allocation. International Journal of Communication Systems, 32(7), 3928.

    Article  Google Scholar 

  24. Wang, J., Hu, J., Min, G., Zhan, W., Ni, Q., & Georgalas, N. (2019). Computation offloading in multi-access edge computing using a deep sequential model based on reinforcement learning. IEEE Communications Magazine, 57(5), 64–69.

    Article  Google Scholar 

  25. Ning, Z., Dong, P., Wang, X., Guo, L., Rodrigues, J. J., Kong, X., et al. (2019). Deep reinforcement learning for intelligent internet of vehicles: An energy-efficient computational offloading scheme. IEEE Transactions on Cognitive Communications and Networking, 5(4), 1060–1072.

    Article  Google Scholar 

  26. Liu, Y., Yu, H., Xie, S., & Zhang, Y. (2019). Deep reinforcement learning for offloading and resource allocation in vehicle edge computing and networks. IEEE Transactions on Vehicular Technology, 68(11), 11158–11168.

    Article  Google Scholar 

  27. Huang, L., Bi, S., & Zhang, Y. J. (2019). Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks. IEEE Transactions on Mobile Computing.

  28. Chen, X., Zhang, H., Wu, C., Mao, S., Ji, Y., & Bennis, M. (2018). Optimized computation offloading performance in virtual edge computing systems via deep reinforcement learning. IEEE Internet of Things Journal

  29. Wang, Y., Wang, K., Huang, H., Miyazaki, T., & Guo, S. (2018). Traffic and computation co-offloading with reinforcement learning in fog computing for industrial applications. IEEE Transactions on Industrial Informatics, 15(2), 976–986.

    Article  Google Scholar 

  30. Xiao, L., Li, Y., Huang, X., & Du, X. (2017). Cloud-based malware detection game for mobile devices with offloading. IEEE Transactions on Mobile Computing, 16(10), 2742–2750.

    Article  Google Scholar 

  31. He, X., Liu, J., Jin, R., Dai, H. (2017). Privacy-aware offloading in mobile-edge computing. In: GLOBECOM 2017-2017 IEEE Global Communications Conference, pp. 1–6. IEEE

  32. Min, M., Wan, X., Xiao, L., Chen, Y., Xia, M., Wu, D., & Dai, H. (2018). Learning-based privacy-aware offloading for healthcare iot with energy harvesting. IEEE Internet of Things Journal, 6(3), 4307–4316.

    Article  Google Scholar 

  33. Elgendy, I. A., Zhang, W., Tian, Y.-C., & Li, K. (2019). Resource allocation and computation offloading with data security for mobile edge computing. Future Generation Computer Systems, 100, 531–541.

    Article  Google Scholar 

  34. Ngo, H. Q., Larsson, E. G., & Marzetta, T. L. (2013). Energy and spectral efficiency of very large multiuser mimo systems. IEEE Transactions on Communications, 61(4), 1436–1449.

    Article  Google Scholar 

  35. Suraweera, H. A., Tsiftsis, T. A., Karagiannidis, G. K., & Nallanathan, A. (2011). Effect of feedback delay on amplify-and-forward relay networks with beamforming. IEEE Transactions on Vehicular Technology, 60(3), 1265–1271.

    Article  Google Scholar 

  36. Ke, H., Wang, J., Wang, H., & Ge, Y. (2019). Joint optimization of data offloading and resource allocation with renewable energy aware for iot devices: A deep reinforcement learning approach. IEEE Access, 7, 179349–179363.

    Article  Google Scholar 

  37. Sutton, R.S., Barto, A.G., et al. (1998). Introduction to Reinforcement Learning vol. 135. MIT press.

  38. Sutton, R.S., McAllester, D.A., Singh, S.P., & Mansour, Y. (2000). Policy gradient methods for reinforcement learning with function approximation. In: Advances in Neural Information Processing Systems, pp. 1057–1063

  39. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., et al. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529.

    Article  Google Scholar 

  40. Van Hasselt, H., Guez, A., & Silver, D. (2016). Deep reinforcement learning with double q-learning. In: Thirtieth AAAI Conference on Artificial Intelligence

  41. Yang, Q., Liu, Y., Chen, T., & Tong, Y. (2019). Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2), 1–19.

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by the National Nature Science Foundation of China (61841602, 61806024), the Jilin Province Education Department Scientific Research Planning Foundation of China (JJKH20210753KJ, JJKH20200618KJ), and the Jilin Province Science and Technology Department Project of China (20190302106GX).

Author information

Authors and Affiliations

  1. School of Computer Technology and Engineering, Changchun Institute of Technology, Changchun, China

    H. C. Ke

  2. National and Local Joint Engineering Research Center for Intelligent Distribution Network Measurement, Control and Safe Operation Technology of National Development and Reform Commission, Changchun Institute of Technology, Changchun, China

    H. C. Ke

  3. College of Computer Science and Engineering, Changchun University of Technology, Changchun, China

    H. Wang & W. J. Sun

  4. College of Computer Science and Technology, Jilin University, Changchun, China

    H. W. Zhao

Authors
  1. H. C. Ke
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. H. Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. H. W. Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. W. J. Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to H. Wang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ke, H.C., Wang, H., Zhao, H.W. et al. Deep reinforcement learning-based computation offloading and resource allocation in security-aware mobile edge computing. Wireless Netw 27, 3357–3373 (2021). https://doi.org/10.1007/s11276-021-02643-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11276-021-02643-w

Keywords

Search

Navigation