Skip to main content
SpringerLink
Log in

Graph convolutional network-based reinforcement learning for tasks offloading in multi-access edge computing

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

To achieve high quality of service for computation-intensive applications, multi-access edge computing (MEC) is proposed for offloading tasks to MEC servers. The emerging reinforcement learning-based task offloading strategies have attracted attention of researchers, but the incomplete Markov models in them result in limited improvements. This work proposes a graph convolutional network-based reinforcement learning (GRL-based) method to enhance the reinforcement learning-based task offloading in MEC. The Graph Convolutional Network is introduced to extract features from tasks through regarding the task set as a directed acyclic graph. Then we construct a complete Markov model for the offloading strategy. In the proposed GRL-based method, the decision process is deployed in the user layer, while the training process is deployed in the cloud layer. An off-policy reinforcement learning method, soft actor-critic, is used to train the offloading strategy, by which the sampling and training can be implemented separately. Several simulation experiments show the proposed GRL-based method performs better than baseline methods, and it can achieve continuous decisions for task offloading efficiently.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Al-Habob AA, Ibrahim A, Dobre OA, Armada AG (2019) Collision-free sequential task offloading for mobile edge computing. IEEE Commun Lett 24(1):71–75

    Article  Google Scholar 

  2. Alameddine HA, Sharafeddine S, Sebbah S, Ayoubi S, Assi C (2019) Dynamic task offloading and scheduling for low-latency IoT services in multi-access edge computing. IEEE J Sel Areas Commun 37(3):668–682

    Article  Google Scholar 

  3. Aliyu M, Murali M, Gital AY, Boukari S (2020) Efficient metaheuristic population-based and deterministic algorithm for resource provisioning using ant colony optimization and spanning tree. Int J Cloud Appl Comput (IJCAC) 10 (2):1–21

    Google Scholar 

  4. Alweshah M, Al Khalaileh S, Gupta BB, Almomani A, Hammouri AI, Al-Betar MA (2020) The monarch butterfly optimization algorithm for solving feature selection problems. Neural Comput Appl 1–15

  5. Andrews JG, Buzzi S, Choi W, Hanly SV, Lozano A, Soong AC, Zhang JC (2014) What will 5G be? IEEE J Sel Areas Commun 32 (6):1065–1082

    Article  Google Scholar 

  6. Babaeizadeh M, Frosio I, Tyree S, Clemons J, Kautz J (2016) Reinforcement learning through asynchronous advantage actor-critic on a GPU. arXiv:161106256

  7. Barto AG, Dietterich TG (2004) Reinforcement learning and its relationship to supervised learning. Handbook Learn Approx Dyn Prog 10:9780470544785

    Google Scholar 

  8. Cao X, Xu J, Zhang R (2018) Mobile edge computing for cellular-connected UAV: Computation offloading and trajectory optimization. In: 2018 IEEE 19Th international workshop on signal processing advances in wireless communications (SPAWC). IEEE, pp 1–5

  9. Chen M, Zhang Y, Li Y, Mao S, Leung VC (2015) EMC: emotion-aware mobile cloud computing in 5G. IEEE Netw 29(2):32–38

    Article  Google Scholar 

  10. Ferrer AJ, Marquès JM, Jorba J (2019) Towards the decentralised cloud: Survey on approaches and challenges for mobile, ad hoc, and edge computing. ACM Comput Surv (CSUR) 51(6):1–36

    Article  Google Scholar 

  11. François-Lavet V, Henderson P, Islam R, Bellemare MG, Pineau J (2018) An introduction to deep reinforcement learning. arXiv:181112560

  12. Gläscher J, Daw N, Dayan P, O’Doherty JP (2010) States versus rewards: dissociable neural prediction error signals underlying model-based and model-free reinforcement learning. Neuron 66(4):585–595

    Article  Google Scholar 

  13. Hämäläinen P, Babadi A, Ma X, Lehtinen J (2020) PPO-CMA: Proximal policy optimization with covariance matrix adaptation. In: 2020 IEEE 30Th international workshop on machine learning for signal processing (MLSP). IEEE, pp 1–6

  14. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  15. Hu H, Wu X, Luo B, Tao C, Xu C, Wu W, Chen Z (2018) Playing 20 question game with policy-based reinforcement learning. arXiv:180807645

  16. Huang L, Bi S, Zhang YJA (2018) Deep reinforcement learning for online offloading in wireless powered mobile-edge computing networks. arXiv:180801977

  17. Kibria MG, Nguyen K, Villardi GP, Zhao O, Ishizu K, Kojima F (2018) Big data analytics, machine learning, and artificial intelligence in next-generation wireless networks. IEEE Access 6:32328–32338

    Article  Google Scholar 

  18. Kipf TN, Welling M (2016) Semi-supervised classification with graph convolutional networks. arXiv:160902907

  19. Khekare G, Verma P, Dhanre U, Raut S, Sheikh S (2020) The optimal path finding algorithm based on reinforcement learning. Int J Softw Sci Comput Intell (IJSSCI) 12(4):1–18

    Article  Google Scholar 

  20. Konda VR, Tsitsiklis JN (2000) Actor-critic algorithms. In: Advances in neural information processing systems, pp 1008–1014

  21. Li D, Deng L, Gupta BB, Wang H, Choi C (2019) A novel CNN based security guaranteed image watermarking generation scenario for smart city applications. Inf Sci 479:432–447

    Article  Google Scholar 

  22. Li J, Gao H, Lv T, Lu Y (2018) Deep reinforcement learning based computation offloading and resource allocation for MEC. In: 2018 IEEE wireless communications and networking conference (WCNC). IEEE, pp 1–6

  23. Liu L, Chang Z, Guo X, Ristaniemi T (2017) Multi-objective optimization for computation offloading in mobile-edge computing. In: 2017 IEEE symposium on computers and communications (ISCC). IEEE, pp 832–837

  24. Lv Z (2020) Security of internet of things edge devices. Softw Pract Exp

  25. Lv Z, Chen D, Wang Q (2020) Diversified technologies in internet of vehicles under intelligent edge computing. IEEE Trans Intell Trans Sys

  26. Lv Z, Chen D, Lou R, Wang Q (2020) Intelligent edge computing based on machine learning for smart city. Futur Gener Comput Syst 115:90–99

    Article  Google Scholar 

  27. Lv Z, Qiao L (2020) Optimization of collaborative resource allocation for mobile edge computing. Comput Commun 161:19–27

    Article  Google Scholar 

  28. Lv Z, Xiu W (2020) Interaction of edge-cloud computing based on SDN and NFV for next generation IoT. IEEE Internet Things J 7(7):5706–5712

    Article  Google Scholar 

  29. Mehrabi M, You D, Latzko V, Salah H, Reisslein M, Fitzek FH (2019) Device-enhanced MEC: Multi-access edge computing (MEC) aided by end device computation and caching: A survey. IEEE Access 7:166079–166108

    Article  Google Scholar 

  30. Memos VA, Psannis KE, Ishibashi Y, Kim BG, Gupta BB (2018) An efficient algorithm for media-based surveillance system (EAMSus) in IoT smart city framework. Futur Gener Comput Syst 83:619–628

    Article  Google Scholar 

  31. Mirri S, Delnevo G, Roccetti M (2020) Is a COVID-19 second wave possible in Emilia-Romagna (Italy)? Forecasting a future outbreak with particulate pollution and machine learning. Computation 8(3):74

    Article  Google Scholar 

  32. Munos R, Stepleton T, Harutyunyan A, Bellemare M (2016) Safe and efficient off-policy reinforcement learning. In: Advances in neural information processing systems, pp 1054–1062

  33. Nachum O, Norouzi M, Xu K, Schuurmans D (2017) Bridging the Gap between value and policy based reinforcement learning. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) Advances in neural information processing systems, vol 30. Curran Associates, Inc., pp 2775–2785

  34. Scarselli F, Gori M, Tsoi AC, Hagenbuchner M, Monfardini G (2008) The graph neural network model. IEEE Trans Neural Netw 20(1):61–80

    Article  Google Scholar 

  35. Tian Y, Fan B, Wu F (2017) L2-net: Deep learning of discriminative patch descriptor in euclidean space. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 661–669

  36. Tran TX, Pompili D (2018) Joint task offloading and resource allocation for multi-server mobile-edge computing networks. IEEE Trans Veh Technol 68(1):856–868

    Article  Google Scholar 

  37. Uva G, Sangiorgio V, Ruggieri S, Fatiguso F (2019) Structural vulnerability assessment of masonry churches supported by user-reported data and modern Internet of Things (IoT). Measurement 131:183–192

    Article  Google Scholar 

  38. Van Brummelen J, O’Brien M, Gruyer D, Najjaran H (2018) Autonomous vehicle perception: The technology of today and tomorrow. Trans Res Part C Emerg Technol S0968090X18302134

  39. Wang J, Hu J, Min G, Zhan W, Ni Q, Georgalas N (2019) Computation offloading in multi-access edge computing using a deep sequential model based on reinforcement learning. IEEE Commun Mag 57(5):64–69

    Article  Google Scholar 

  40. Yao L, Mao C, Luo Y (2019) Graph convolutional networks for text classification. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 7370–7377

  41. Zhang K, Zhu Y, Leng S, He Y, Maharjan S, Zhang Y (2019) Deep learning empowered task offloading for mobile edge computing in urban informatics. IEEE Internet Things J 6(5):7635–7647

    Article  Google Scholar 

  42. Zhang Y, Zhao Z, Shu C, Min G, Wang Z (2018) Embedding virtual network functions with backup for reliable large-scale edge computing. In: 2018 5Th IEEE international conference on cyber security and cloud computing (CSCloud)/2018 4th IEEE international conference on edge computing and scalable cloud (EdgeCom). IEEE, pp 190–195

  43. Zhao X, Yang K, Chen Q, Peng D, Jiang H, Xu X, Shuang X (2019) Deep learning based mobile data offloading in mobile edge computing systems. Futur Gener Comput Syst 99:346–355

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by National Natural Science Foundation of China under Grant 62076202, 61976178.

Author information

Authors and Affiliations

  1. School of Computer Science, Northwestern Polytechnical University, Xi’an, Shaanxi Province, 710072, China

    Lixiong Leng, Jingchen Li, Haobin Shi & Yi’an Zhu

Authors
  1. Lixiong Leng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jingchen Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haobin Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yi’an Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haobin Shi.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Leng, L., Li, J., Shi, H. et al. Graph convolutional network-based reinforcement learning for tasks offloading in multi-access edge computing. Multimed Tools Appl 80, 29163–29175 (2021). https://doi.org/10.1007/s11042-021-11130-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-021-11130-5

Keywords

Search

Navigation