Skip to main content

Advertisement

SpringerLink
Log in

Joint computation offloading and task caching for multi-user and multi-task MEC systems: reinforcement learning-based algorithms

  • Published:
Wireless Networks Aims and scope Submit manuscript

Abstract

Computation offloading at mobile edge computing (MEC) servers can mitigate the resource limitation and reduce the communication latency for mobile devices. Thereby, in this study, we proposed an offloading model for a multi-user MEC system with multi-task. In addition, a new caching concept is introduced for the computation tasks, where the application program and related code for the completed tasks are cached at the edge server. Furthermore, an efficient model of task offloading and caching integration is formulated as a nonlinear problem whose goal is to reduce the total overhead of time and energy. However, solving these types of problems is computationally prohibitive, especially for large-scale of mobile users. Thus, an equivalent form of reinforcement learning is created where the state spaces are defined based on all possible solutions and the actions are defined on the basis of movement between the different states. Afterwards, two effective Q-learning and Deep-Q-Network-based algorithms are proposed to derive the near-optimal solution for this problem. Finally, experimental evaluations verify that our proposed model can substantially minimize the mobile devices’ overhead by deploying computation offloading and task caching strategy reasonably.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. The requests’ number of computation task j is assumed to be known [26].

  2. Deep Q Network is only used in Fig. 6(b) in which the number of devices is increasing and it is becomes difficult with Q table for computing and storing the corresponding Q value as mentioned above.

References

  1. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., et al. (2016). Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467.

  2. Abd El-Latif, A. A., Abd-El-Atty, B., Mazurczyk, W., Fung, C., & Venegas-Andraca, S. E. (2020). Secure data encryption based on quantum walks for 5G internet of things scenario. IEEE Transactions on Network and Service Management, 17(1), 118–131.

    Google Scholar 

  3. Abd El-Latif, A. A., Abd-El-Atty, B., Venegas-Andraca, S. E., Elwahsh, H., Piran, M. J., Bashir, A. K., et al. (2020). Providing end-to-end security using quantum walks in IoT networks. IEEE Access, 8, 92687–92696.

    Google Scholar 

  4. Abd EL-Latif, A. A., Abd-El-Atty, B., Venegas-Andraca, S. E. and Mazurczyk, W., (2019). Efficient quantum-based security protocols for information sharing and data protection in 5G networks. Future Generation Computer Systems, 100, 893–906.

    Google Scholar 

  5. Abou-Nassar, E. M., Iliyasu, A. M., El-Kafrawy, P. M., Song, O.-Y., Bashir, A. K., & Abd El-Latif, A. A. (2020). Ditrust chain: Towards blockchain-based trust models for sustainable healthcare iot systems. IEEE Access, 8, 111223–111238.

    Google Scholar 

  6. Ale, L., Zhang, N., Wu, H., Chen, D., & Han, T. (2019). Online proactive caching in mobile edge computing using bidirectional deep recurrent neural network. IEEE Internet of Things Journal, 6, 5520–5530.

    Google Scholar 

  7. AlZu’bi, S., Shehab, M., Al-Ayyoub, M., Jararweh, Y., & Gupta, B. (2020). Parallel implementation for 3D medical volume fuzzy segmentation. Pattern Recognition Letters, 130, 312–318.

    Google Scholar 

  8. Ananthanarayanan, G., Bahl, V., Cox, L., Crown, A., Nogbahi, S., & Shu, Y. (2019). Video analytics-killer app for edge computing. In Proceedings of the 17th annual international conference on mobile systems, applications, and services (pp. 695–696). ACM.

  9. Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017). A brief survey of deep reinforcement learning. arXiv preprint arXiv:1708.05866.

  10. Boukerche, A., Guan, S., & Grande, R. E. D. (2019). Sustainable offloading in mobile cloud computing: Algorithmic design and implementation. ACM Computing Surveys (CSUR), 52(1), 1–37.

    Google Scholar 

  11. Chen, J., & Ran, X. (2019). Deep learning with edge computing: A review. Proceedings of the IEEE, 107(8), 1655–1674.

    Google Scholar 

  12. Chen, L., Qu, H., Zhao, J., Chen, B., & Principe, J. C. (2016). Efficient and robust deep learning with correntropy-induced loss function. Neural Computing and Applications, 27(4), 1019–1031.

    Google Scholar 

  13. Chen, M.-H., Liang, B., & Dong, M. (2017). Joint offloading and resource allocation for computation and communication in mobile cloud with computing access point. In IEEE INFOCOM 2017-IEEE conference on computer communications (pp. 1–9). IEEE.

  14. Chen, X. (2014). Decentralized computation offloading game for mobile cloud computing. IEEE Transactions on Parallel and Distributed Systems, 26(4), 974–983.

    Google Scholar 

  15. Chen, X., Jiao, L., Li, W., & Fu, X. (2015). Efficient multi-user computation offloading for mobile-edge cloud computing. IEEE/ACM Transactions on Networking, 24(5), 2795–2808.

    Google Scholar 

  16. Chen, Y., Zhang, N., Zhang, Y., Chen, X., Wu, W., & Shen, X. S. (2019). Toffee: Task offloading and frequency scaling for energy efficiency of mobile devices in mobile edge computing. IEEE Transactions on Cloud Computing, https://doi.org/10.1109/TCC.2019.2923692.

    Article  Google Scholar 

  17. Deb, S., & Monogioudis, P. (2015). Learning-based uplink interference management in 4G LTE cellular systems. IEEE/ACM Transactions on Networking, 23(2), 398–411.

    Google Scholar 

  18. Elgendy, I. A., Zhang, W., Tian, Y.-C., & Li, K. (2019). Resource allocation and computation offloading with data security for mobile edge computing. Future Generation Computer Systems, 100, 531–541.

    Google Scholar 

  19. Elgendy, I. A., Zhang, W.-Z., Zeng, Y., He, H., Tian, Y.-C., & Yang, Y. (2020). Efficient and secure multi-user multi-task computation offloading for mobile-edge computing in mobile IoT networks. IEEE Transactions on Network and Service Management, 17, 2410–2422.

    Google Scholar 

  20. Elgendy, I., Zhang, W., Liu, C., & Hsu, C.-H. (2018). An efficient and secured framework for mobile cloud computing. IEEE Transactions on Cloud Computing. https://doi.org/10.1109/TCC.2018.2847347

    Article  Google Scholar 

  21. Fooladivanda, D., & Rosenberg, C. (2011). Joint resource allocation and user association for heterogeneous wireless cellular networks. IEEE Transactions on Wireless Communications, 12(1), 248–257.

    Google Scholar 

  22. Gad, R., Talha, M., Abd El-Latif, A. A., Zorkany, M., Ayman, E.-S., Nawal, E.-F., et al. (2018). Iris recognition using multi-algorithmic approaches for cognitive internet of things (CIoT) framework. Future Generation Computer Systems, 89, 178–191.

    Google Scholar 

  23. Guo, S., Chen, M., Liu, K., Liao, X., & Xiao, B. (2020). Robust computation offloading and resource scheduling in cloudlet-based mobile cloud computing. IEEE Transactions on Mobile Computing. https://doi.org/10.1109/TMC.2020.2973993

    Article  Google Scholar 

  24. Gupta, B., & Quamara, M. (2020). An overview of internet of things (IoT): Architectural aspects, challenges, and protocols. Concurrency and Computation: Practice and Experience, 32(21), e4946.

    Google Scholar 

  25. Han, Y., Wang, X., Leung, V., Niyato, D., Yan, X., & Chen, X. (2019). Convergence of edge computing and deep learning: A comprehensive survey. arXiv preprint arXiv:1907.08349.

  26. Hao, Y., Chen, M., Hu, L., Hossain, M. S., & Ghoneim, A. (2018). Energy efficient task caching and offloading for mobile edge computing. IEEE Access, 6, 11365–11373.

    Google Scholar 

  27. Huang, L., Feng, X., Zhang, C., Qian, L., & Wu, Y. (2019). Deep reinforcement learning-based joint task offloading and bandwidth allocation for multi-user mobile edge computing. Digital Communications and Networks, 5(1), 10–17.

    Google Scholar 

  28. Jararweh, Y., Doulat, A., Alqudah, O., Ahmed, E., Al-Ayyoub, M., & Benkhelifa, E. (2016). The future of mobile cloud computing: Integrating cloudlets and mobile edge computing. In International conference on telecommunications.

  29. Khayyat, M., Alshahrani, A., Alharbi, S., Elgendy, I. A., Paramonov, A., & Koucheryavy, A. (2020). Multilevel service-provisioning-based autonomous vehicle applications. Sustainability, 12(6), 2497–2513.

    Google Scholar 

  30. Khayyat, M., Elgendy, I. A., Muthanna, A., Alshahrani, A. S., Alharbi, S., & Koucheryavy, A. (2020). Advanced deep learning-based computational offloading for multilevel vehicular edge-cloud computing networks. IEEE Access, 8, 137052–137062.

    Google Scholar 

  31. Li, D., Deng, L., Gupta, B. B., Wang, H., & Choi, C. (2019). A novel cnn based security guaranteed image watermarking generation scenario for smart city applications. Information Sciences, 479, 432–447.

    Google Scholar 

  32. Li, J., Gao, H., Lv, T., & Lu, Y. (2018). Deep reinforcement learning based computation offloading and resource allocation for MEC. In 2018 IEEE Wireless communications and networking conference (WCNC) (pp. 1–6). IEEE.

  33. Lin, X., Wang, Y., Xie, Q., & Pedram, M. (2015). Task scheduling with dynamic voltage and frequency scaling for energy minimization in the mobile cloud computing environment. IEEE Transactions on Services Computing, 8(2), 175–186.

    Google Scholar 

  34. Liu, P., Xu, G., Yang, K., Wang, K., & Meng, X. (2018). Jointly optimized energy-minimal resource allocation in cache-enhanced mobile edge computing systems. IEEE Access, 7, 3336–3347.

    Google Scholar 

  35. Liu, Y., Peng, J., Kang, J., Iliyasu, A. M., Niyato, D., & El-Latif, A. A. A. (2020). A secure federated learning framework for 5G networks. arXiv preprint arXiv:2005.05752.

  36. Luong, N. C., Hoang, D. T., Gong, S., Niyato, D., Wang, P., Liang, Y.-C., et al. (2019). Applications of deep reinforcement learning in communications and networking: A survey. IEEE Communications Surveys & Tutorials, 21, 3133–3174.

    Google Scholar 

  37. Lyu, X., Ren, C., Ni, W., Tian, H., Liu, R. P., & Tao, X. (2020). Distributed online learning of cooperative caching in edge cloud. IEEE Transactions on Mobile Computing. https://doi.org/10.1109/TMC.2020.2983924

    Article  Google Scholar 

  38. Lyu, X., & Tian, H. (2016). Adaptive receding horizon offloading strategy under dynamic environment. IEEE Communications Letters, 20(5), 878–881.

    Google Scholar 

  39. Mach, P., & Becvar, Z. (2017). Mobile edge computing: A survey on architecture and computation offloading. IEEE Communications Surveys and Tutorials, 19(3), 1628–1656.

    Google Scholar 

  40. Mao, Y., Zhang, J., Song, S., & Letaief, K. B. (2016). Power-delay tradeoff in multi-user mobile-edge computing systems. In 2016 IEEE Global communications conference (GLOBECOM) (pp. 1–6). IEEE.

  41. Min, M., Xiao, L., Chen, Y., Cheng, P., Wu, D., & Zhuang, W. (2019). Learning-based computation offloading for IoT devices with energy harvesting. IEEE Transactions on Vehicular Technology, 68(2), 1930–1941.

    Google Scholar 

  42. Nur, F. N., Islam, S., Moon, N. N., Karim, A., Azam, S., & Shanmugam, B. (2019). Priority-based offloading and caching in mobile edge cloud. Journal of Communications Software and Systems, 15(2), 193–201.

    Google Scholar 

  43. Rappaport, T. S. (2009). Wireless communications: Principles and practice. London: Prentice Hall.

    MATH  Google Scholar 

  44. Ruder, S. (2016). An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747.

  45. Sadeghi, A., Sheikholeslami, F., & Giannakis, G. B. (2017). Optimal and scalable caching for 5G using reinforcement learning of space-time popularities. IEEE Journal of Selected Topics in Signal Processing, 12(1), 180–190.

    Google Scholar 

  46. Stergiou, C. L., Psannis, K. E., & Gupta, B. B. (2020). IoT-based big data secure management in the fog over a 6G wireless network. IEEE Internet of Things Journal. https://doi.org/10.1109/JIOT.2020.3033131

    Article  Google Scholar 

  47. Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction. Cambridge: MIT Press.

    MATH  Google Scholar 

  48. Tewari, A., & Gupta, B. (2020). Security, privacy and trust of different layers in internet-of-things (IoTs) framework. Future Generation Computer Systems, 108, 909–920.

    Google Scholar 

  49. Vallina-Rodriguez, N., & Crowcroft, J. (2013). Energy management techniques in modern mobile handsets. IEEE Communications Surveys Tutorials, 15(1), 179–198.

    Google Scholar 

  50. Wang, H., Li, R., Fan, L., & Zhang, H. (2017). Joint computation offloading and data caching with delay optimization in mobile-edge computing systems. In International conference on wireless communications and signal processing (pp. 1–6).

  51. Wang, H., Li, Z., Li, Y., Gupta, B., & Choi, C. (2020). Visual saliency guided complex image retrieval. Pattern Recognition Letters, 130, 64–72.

    Google Scholar 

  52. Wang, S., Zhang, X., Zhang, Y., Wang, L., Yang, J., & Wang, W. (2017). A survey on mobile edge networks: Convergence of computing, caching and communications. IEEE Access, 5(99), 6757–6779.

    Google Scholar 

  53. Wen, Y., Zhang, W., & Luo, H. (2012). Energy-optimal mobile application execution: Taming resource-poor mobile devices with cloud clones. In IEEE INFOCOM (pp. 2716–2720).

  54. Wang, X., Han, Y., Leung, V. C., Niyato, D., Yan, X., & Chen, X. (2020). Convergence of edge computing and deep learning: A comprehensive survey. IEEE Communications Surveys & Tutorials, 22, 869–904.

    Google Scholar 

  55. Yadav, R., Zhang, W., Kaiwartya, O., Singh, P. R., Elgendy, I. A., & Tian, Y.-C. (2018). Adaptive energy-aware algorithms for minimizing energy consumption and sla violation in cloud computing. IEEE Access, 6, 55923–55936.

    Google Scholar 

  56. Yang, P., Zhang, N., Zhang, S., Yu, L., Zhang, J., & Shen, X. (2019). Content popularity prediction towards location-aware mobile edge caching. IEEE Transactions on Multimedia, 21(4), 915–929.

    Google Scholar 

  57. Yi, C., Cai, J., & Su, Z. (2020). A multi-user mobile computation offloading and transmission scheduling mechanism for delay-sensitive applications. IEEE Transactions on Mobile Computing, 19(1), 29–43.

    Google Scholar 

  58. Zhang, W., Wen, Y., & Wu, D. O. (2014). Collaborative task execution in mobile cloud computing under a stochastic wireless channel. IEEE Transactions on Wireless Communications, 14(1), 81–93.

    Google Scholar 

  59. Zhang, W.-Z., Elgendy, I. A., Hammad, M., Iliyasu, A. M., Du, X., Guizani, M., et al. (2020). Secure and optimized load balancing for multi-tier IoT and edge-cloud computing systems. IEEE Internet of Things Journal. https://doi.org/10.1109/JIOT.2020.3042433

    Article  Google Scholar 

  60. Zhu, H., Cao, Y., Wang, W., Jiang, T., & Jin, S. (2018). Deep reinforcement learning for mobile edge caching: Review, new features, and open issues. IEEE Network, 32(6), 50–57.

    Google Scholar 

Download references

Acknowledgements

This work was supported in part by the Key Research and Development Program for Guangdong Province (2019B010136001) and the National Key Research and Development Plan under Grant 2017YFB0801801, in part by the National Natural Science Foundation of China (NSFC) under Grants 61672186 and 61872110, in part by Shenzhen Science and Technology Research and Development Fundation (JCYJ20190806143418198).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China

    Ibrahim A. Elgendy, Wei-Zhe Zhang & Hui He

  2. Department of Computer Science, Faculty of Computers and Information, Menoufia University, Shebin El-Koom, 32511, Egypt

    Ibrahim A. Elgendy

  3. Peng Cheng Laboratory, Cyberspace Security Research Center, Shenzhen, China

    Wei-Zhe Zhang

  4. National Institute of Technology Kurukshetra, Kurukshetra, India

    Brij B. Gupta

  5. Asia University, Taichung City, Taiwan

    Brij B. Gupta

  6. Mathematics and Computer Science Department, Faculty of Science, Menoufia University, P.O. Box 32511, Shebin El-Koom, Egypt

    Ahmed A. Abd El-Latif

Authors
  1. Ibrahim A. Elgendy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wei-Zhe Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hui He
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Brij B. Gupta
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ahmed A. Abd El-Latif
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ibrahim A. Elgendy.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Elgendy, I.A., Zhang, WZ., He, H. et al. Joint computation offloading and task caching for multi-user and multi-task MEC systems: reinforcement learning-based algorithms. Wireless Netw 27, 2023–2038 (2021). https://doi.org/10.1007/s11276-021-02554-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11276-021-02554-w

Keywords

Search

Navigation