Skip to main content
SpringerLink
Log in

A Clustering Offloading Decision Method for Edge Computing Tasks Based on Deep Reinforcement Learning

  • Published:
New Generation Computing Aims and scope Submit manuscript

Abstract

In many IoT scenarios, the resources of terminal devices are limited, and it is difficult to provide services with low latency and low energy consumption. Mobile edge computing is an effective solution by offloading computing tasks to edge server processing. There are some problems in the existing offloading decision algorithms: the offloading decision method based on heuristic algorithms cannot dynamically adjust the policy in the changing environment; the offloading algorithm based on deep reinforcement learning will lead to slow convergence and poor exploration effect due to the problem of dimension explosion. To solve the above problems, this paper designs an offloading decision algorithm to make dynamic decisions in a mobile edge computing network with multi-device access. The algorithm comprehensively considers the energy consumption of terminal equipment, offloading overhead, average delay and success rate of task completion, aiming to achieve the highest total revenue of the whole system in a period of time. In this work, the online offloading problem is abstracted as a Markov decision process. Based on the Double Dueling Deep Q-Network (D3QN) algorithm, the offloading decision is designed to adapt to the highly dynamic environment of the edge computing network and solve the problem of high state space complexity. In addition, this paper innovatively introduces a clustering algorithm into deep reinforcement learning (DRL) to preprocess the action space and solve the explosion problem of the action space dimension caused by the increase of terminal devices. The experimental results show that the proposed algorithm is superior to the baseline strategies such as Deep Q-Network (DQN) algorithm in convergence speed and total reward.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

  1. He, X.F., Jin, R.C., Dai, H.Y.: Deep PDS-learning for privacy-aware offloading in MEC-enabled IoT. IEEE Internet Things J. 6(3), 4547–4555 (2019)

    Article  Google Scholar 

  2. Mansouri, Y., Babar, M.A.: A review of edge computing: features and resource virtualization. J. Parallel Distrib. Comput. 150, 155–183 (2021)

    Article  Google Scholar 

  3. Chen, Y., Zhang, N., Zhang, Y.C., Chen, X.: Dynamic computation offloading in edge computing for internet of things. IEEE Internet Things J. 6(3), 4242–4251 (2019)

    Article  Google Scholar 

  4. Jiang, C.F., Cheng, X.L., Gao, H.H., Zhou, X., Wan, J.: Toward computation offloading in edge computing: a survey. IEEE Access 7, 131543–131558 (2019)

    Article  Google Scholar 

  5. Islam, A., Debnath, A., Ghose, M., Chakraborty, S.: A survey on task offloading in multi-access edge computing. J. Syst. Architect. 118, 16 (2021)

    Article  Google Scholar 

  6. Mao, Y.Y., You, C.S., Zhang, J., Huang, K.B., Letaief, K.B.: A survey on mobile edge computing: the communication perspective. IEEE Commun. Surv. Tutorials 19(4), 2322–2358 (2017)

    Article  Google Scholar 

  7. Zhang, J., Xia, W.W., Yan, F., Shen, L.F.: Joint computation offloading and resource allocation optimization in heterogeneous networks with mobile edge computing. IEEE Access 6, 19324–19337 (2018)

    Article  Google Scholar 

  8. Lin, H., Zeadally, S., Chen, Z.H., Labiod, H., Wang, L.S.: A survey on computation offloading modeling for edge computing. J. Netw. Comput. Appl. 169, 25 (2020)

    Article  Google Scholar 

  9. Cui, L.Z., Chen, Z.T., Yang, S., Ming, Z.X., Li, Q., Zhou, Y.P., Chen, S.P., Lu, Q.H.: A blockchain-based containerized edge computing platform for the internet of vehicles. IEEE Internet Things J. 8(4), 2395–2408 (2021)

    Article  Google Scholar 

  10. Huang, D., Wang, P., Niyato, D.: A dynamic offloading algorithm for mobile computing. IEEE Trans. Wireless Commun. 11(6), 1991–1995 (2012)

    Article  Google Scholar 

  11. You, C.S., Huang, K.B., Chae, H., Kim, B.H.: Energy-efficient resource allocation for mobile-edge computation offloading. IEEE Trans. Wireless Commun. 16(3), 1397–1411 (2017)

    Article  Google Scholar 

  12. Ebrahimzadeh, A., Maier, M.: Distributed cooperative computation offloading in multi-access edge computing fiber-wireless networks. Opt. Commun. 452, 130–139 (2019)

    Article  Google Scholar 

  13. Ale, L., Zhang, N., Wu, H.C., Chen, D.J., Han, T.: Online proactive caching in mobile edge computing using bidirectional deep recurrent neural network. IEEE Internet Things J. 6(3), 5520–5530 (2019)

    Article  Google Scholar 

  14. Kober, J., Bagnell, J.A., Peters, J.: Reinforcement learning in robotics: a survey. Int. J. Robot. Res. 32(11), 1238–1274 (2013)

    Article  Google Scholar 

  15. Chen, M., Hao, Y.X.: Task offloading for mobile edge computing in software defined ultra-dense network. IEEE J. Sel. Areas Commun. 36(3), 587–597 (2018)

    Article  MathSciNet  Google Scholar 

  16. Chen, M.H., Dong, M., Liang, B.: Resource sharing of a computing access point for multi-user mobile cloud offloading with delay constraints. IEEE Trans. Mob. Comput. 17(12), 2868–2881 (2018)

    Article  Google Scholar 

  17. Al-Asadi, M.A., Tasdemir, S.: Empirical comparisons for combining balancing and feature selection strategies for characterizing football players using FIFA video game system. IEEE Access 9, 149266–149286 (2021)

    Article  Google Scholar 

  18. Qiu, X.Y., Liu, L.B., Chen, W.H., Hong, Z.C., Zheng, Z.B.: Online deep reinforcement learning for computation offloading in blockchain-empowered mobile edge computing. IEEE Trans. Veh. Technol. 68(8), 8050–8062 (2019)

    Article  Google Scholar 

  19. Zhao, R., Wang, X.J., Xia, J.J., Fan, L.S.: Deep reinforcement learning based mobile edge computing for intelligent internet of things. Phys. Commun. 43, 7 (2020)

    Article  Google Scholar 

  20. Wang, H.N., Liu, N., Zhang, Y.Y., Feng, D.W., Huang, F., Li, D.S., Zhang, Y.M.: Deep reinforcement learning: a survey. Front. Inf. Technol. Electr. Eng. 21(12), 1726–1744 (2020)

    Article  Google Scholar 

  21. Chen, Z., Wang, X.D.: Decentralized computation offloading for multi-user mobile edge computing: a deep reinforcement learning approach. EURASIP J. Wirel. Commun. Netw. 2020(1), 21 (2020)

    Article  Google Scholar 

  22. Zhang, Y., Yao, J.G., Guan, H.B.: Intelligent cloud resource management with deep reinforcement learning. IEEE Cloud Comput. 4(6), 60–69 (2017)

    Article  Google Scholar 

  23. Zhang, C., Zheng, Z.X.: Task migration for mobile edge computing using deep reinforcement learning. Future Gener. Comput. Syst. Int. J. Esci. 96, 111–118 (2019)

    Article  Google Scholar 

  24. Chen, W.H., Liu, B.C., Huang, H.W., Guo, S., Meng, Z.B.: When UAV swarm meets edge-cloud computing: the QoS perspective. IEEE Netw. 33(2), 36–43 (2019)

    Article  Google Scholar 

  25. Liu, M.T., Yu, F.R., Teng, Y.L., Leung, V.C.M., Song, M.: Computation offloading and content caching n wireless blockchain networks with mobile edge computing. IEEE Trans. Veh. Technol. 67(11), 11008–11021 (2018)

    Article  Google Scholar 

  26. Chen, W.H., Yaguchi, Y., Naruse, K., Watanobe, Y., Nakamura, K.: QoS-aware robotic streaming workflow allocation in cloud robotics systems. IEEE Trans. Serv. Comput. 14(2), 544–558 (2021)

    Article  Google Scholar 

  27. Huang, X.Y., Leng, S.P., Maharjan, S., Zhang, Y.: Multi-agent deep reinforcement learning for computation offloading and interference coordination in small cell networks. IEEE Trans. Veh. Technol. 70(9), 9282–9293 (2021)

    Article  Google Scholar 

  28. Rioul, O., Magossi, J.C.: On Shannon’s formula and Hartley’s rule: beyond the mathematical coincidence. Entropy 16(9), 4892–4910 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  29. Seng, D.W., Zhang, J.M., Shi, X.Y.: Visual analysis of deep Q-network. KSII Trans. Internet Inf. Syst. 15(3), 853–873 (2021)

    Google Scholar 

  30. Zhang, R., Yu, F.R., Liu, J., Huang, T., Liu, Y.J.: Deep reinforcement learning (DRL)-based device-to-device (D2D) caching With blockchain and mobile edge computing. IEEE Trans. Wireless Commun. 19(10), 6469–6485 (2020)

    Article  Google Scholar 

  31. Shi, C.M., Wei, B.T., Wei, S.L., Wang, W., Liu, H., Liu, J.L.: A quantitative discriminant method of elbow point for the optimal number of clusters in clustering algorithm. EURASIP J. Wirel. Commun. Netw. 2021(1), 16 (2021)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Physics and Electronic Engineering, Sichuan Normal University, 610101, Chengdu, Sichuan, China

    Zhen Zhang, Huanzhou Li, Zhangguo Tang, Dinglin Gu & Jian Zhang

  2. Institute of Cyber and Communication Technology, Sichuan Normal University, 610101, Chengdu, Sichuan, China

    Zhen Zhang, Huanzhou Li, Zhangguo Tang, Dinglin Gu & Jian Zhang

Authors
  1. Zhen Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Huanzhou Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhangguo Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Dinglin Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jian Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jian Zhang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Z., Li, H., Tang, Z. et al. A Clustering Offloading Decision Method for Edge Computing Tasks Based on Deep Reinforcement Learning. New Gener. Comput. 41, 85–108 (2023). https://doi.org/10.1007/s00354-022-00199-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00354-022-00199-7

Keywords

Search

Navigation