Skip to main content

Advertisement

Springer Nature Link
Log in

Data transmission optimization in edge computing using multi-objective reinforcement learning

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Reducing network energy consumption and balancing workload are two key optimization goals for data transmission in edge computing field. However, these two goals are likely to be conflicting in some cases and fail to achieve the optimum simultaneously. In this paper, we design a new data transmission optimization algorithm using multi-objective reinforcement learning. We design the vector of rewards for the two objectives, and update Pareto approximate set by multiple state steps to approach the optimal solution. In every step, we classify the candidate links into four different levels for path selection. We aggregate network traffic to construct minimum topology subset, minimizing the number of occupied device to reduce energy consumption. We optimize the load distribution on those selected links, minimizing maximum congestion factor to balance workload. For action selection, we leverage roulette-based Chebyshev scalarization function to solve the weight selection problem for multi-objectives and enforce exploration to avoid falling into local optimum. To improve the convergence rate, we design heuristic factor to control the search of solution space and enhance the guiding effect of the existing optimal solution. Simulation result shows that the proposed algorithm achieves good performance in energy-saving and load balance at the same time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Algorithm 2
Algorithm 3
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data availability

All authors confirm that the data supporting the findings of this study are available within the article.

References

  1. Taheri S, Ahmadi A, Mohammadi-ivatloo B, Asadi S (2021) Fault detection diagnostic for HVAC systems via deep learning algorithms. Energy Build 250:111275

    Article  Google Scholar 

  2. McEnroe P, Wang S, Liyanage M (2022) A survey on the convergence of edge computing and AI for UAVs: opportunities and challenges. IEEE Internet Things J 9:15435–15459

    Article  Google Scholar 

  3. Hu L, Miao Y, Wu G, Hassan MM, Humar I (2019) irobot-factory: an intelligent robot factory based on cognitive manufacturing and edge computing. Future Gener Comput Syst 90:569–577

    Article  Google Scholar 

  4. Zhang R, Shu H, Navaei YD (2022) Load balancing in edge computing using integer linear programming based genetic algorithm and multilevel control approach. Wirel Commun Mobile Comput 19:15435–15459

    Google Scholar 

  5. Chen W, Liu B, Huang H, Guo S, Zheng Z (2019) When UAV swarm meets edge-cloud computing: the QoS perspective. IEEE Netw 33:36–43

    Google Scholar 

  6. Wang X, Li J, Ning Z, Song Q, Guo L, Guo S, Obaidat MS (2023) Wireless powered mobile edge computing networks: a survey. ACM Comput Surv 55:1–37

    Google Scholar 

  7. Raeisi-Varzaneh M, Dakkak O, Habbal A, Kim BS (2023) Resource scheduling in edge computing: architecture, taxonomy, open issues and future research directions. IEEE Access 11:25329–25350

    Article  Google Scholar 

  8. Montazerolghaem A (2021) Software-defined internet of multimedia things: energy-efficient and load-balanced resource management. IEEE Internet Things J 9:2432–2442

    Article  Google Scholar 

  9. Yan L, Chen H, Tu Y, Zhou X (2022) A task offloading algorithm with cloud edge jointly load balance optimization based on deep reinforcement learning for unmanned surface vehicles. IEEE Access 10:16566–16576

    Article  Google Scholar 

  10. Hu N, Xiang M, Huang C, Qin L, Yang B, Wang R, Luo Z (2022) An efficient computing task offloading strategy based on energy consumption and load balancing degree. In: 2022 4th International Academic Exchange Conference on Science and Technology Innovation (IAECST), pp 860–866

  11. Wang Z, Rong H, Jiang H, Xiao Z, Zeng F (2022) A load-balanced and energy-efficient navigation scheme for UAV-mounted mobile edge computing. IEEE Trans Netw Sci Eng 9:3659–3674

    Article  MathSciNet  Google Scholar 

  12. Perin G, Berno M, Erseghe T, Rossi M (2022) Towards sustainable edge computing through renewable energy resources and online, distributed and predictive scheduling. IEEE Trans Netw Serv Manag 19:306–321

    Article  Google Scholar 

  13. Ma L, Cui X, Li Y (2023) Load balancing and energy saving algorithm based on deep q-learning in mobile edge computing. In: 2023 35th Chinese Control and Decision Conference (CCDC), pp 3736–3741

  14. Li Z, Yu K, Zhou H, Wu X (2023) Dqn-based collaborative computation offloading for edge load balancing. In: 2023 8th IEEE International Conference on Network Intelligence and Digital Content (IC-NIDC), pp 01–06

  15. Long S, Zhang Y, Deng Q, Pei T, Ouyang J, Xia Z (2023) An efficient task offloading approach based on multi-objective evolutionary algorithm in cloud-edge collaborative environment. IEEE Trans Netw Sci Eng 10:645–657

    Article  Google Scholar 

  16. Yan J, Wang H, Li X, Yi S, Qin Y (2020) Multi-objective disaster backup in inter-datacenter using reinforcement learning. In: Wireless Algorithms, Systems, and Applications

  17. Yi S, Li X, Wang H, Qin Y, Yan J (2021) Energy-aware disaster backup among cloud datacenters using multiobjective reinforcement learning in software defined network. Concurr Comput Pract Exp 34:e6588

    Article  Google Scholar 

  18. Yu M, Wang C, Liu H, Li X, Wang X, Wang H (2022) An energy-aware network routing algorithm based on Q-learning. In: 2022 International Conference on High Performance Big Data and Intelligent Systems (HDIS), pp 254–258

  19. Priyadarsini M, Bera P (2021) Software defined networking architecture, traffic management, security, and placement: a survey. Comput Netw 192:108047

    Article  Google Scholar 

  20. Tüysüz MF, Ankarali ZK, Gözüpek D (2017) A survey on energy efficiency in software defined networks. Comput Netw 113:188–204

    Article  Google Scholar 

  21. Chen Y-R, Rezapour A, Tzeng W-G, Tsai S-C (2020) RL-routing: an SDN routing algorithm based on deep reinforcement learning. IEEE Trans Netw Sci Eng 7:3185–3199

    Article  Google Scholar 

  22. Das S, Panda KG, Sen D, Arif W (2021) Maximizing last-minute backup in endangered time-varying inter-datacenter networks. IEEE/ACM Trans Netw 29:2646–2663

    Article  Google Scholar 

  23. Prete L, Farina F, Campanella M, Biancini A (2012) Energy efficient minimum spanning tree in openflow networks. In: 2012 European Workshop on Software Defined Networking, pp 36–41

  24. Mittal R, Lam VT, Dukkipati N, Blem ER, Wassel HMG, Ghobadi M, Vahdat A, Wang Y, Wetherall D, Zats D (2015) Timely: RTT-based congestion control for the datacenter. In: Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication

  25. Wei Y, Zhang X, Xie L, Leng S (2016) Energy-aware traffic engineering in hybrid SDN/ip backbone networks. J Commun Netw 18(4):559–566

    Article  Google Scholar 

  26. Gao Y, Wang H, Zhu R, Yi S, Gao C, Huang F (2015) Minimizing energy consumption with a cloneant-based routing algorithm for communication network. In: 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security, and 2015 IEEE 12th International Conference on Embedded Software and Systems, pp 521–526

  27. Wang Y, Su S, Liu AX, Zhang Z (2014) Multiple bulk data transfers scheduling among datacenters. Comput Netw 68:123–137

    Article  Google Scholar 

  28. Zhu R, Wang H, Gao Y, Yi S, Zhu F (2015) Energy saving and load balancing for SDN based on multi-objective particle swarm optimization. In: International Conference on Algorithms and Architectures for Parallel Processing

  29. Liu H, Li Z, Huang K, Wang R, Cheng G, Li T-X (2023) Evolutionary reinforcement learning algorithm for large-scale multi-agent cooperation and confrontation applications. J Supercomput 80:2319–2346

    Article  Google Scholar 

  30. Zhao X, Ding S, An Y, Jia W (2018) Applications of asynchronous deep reinforcement learning based on dynamic updating weights. Appl Intell 49:581–591

    Article  Google Scholar 

  31. Kröse BJA (1995) Learning from delayed rewards. Robot Auton Syst 15:233–235

    Article  Google Scholar 

  32. Yao Z, Zhang G, Lu D, Liu H (2019) Data-driven crowd evacuation: a reinforcement learning method. Neurocomputing 366:314–327

    Article  Google Scholar 

  33. Li X (2021) An efficient data evacuation strategy using multi-objective reinforcement learning. Appl Intell 52:7498–7512

    Article  Google Scholar 

  34. Li M, Yang S, Liu X (2016) Pareto or non-pareto: bi-criterion evolution in multiobjective optimization. IEEE Trans Evolut Comput 20:645–665

    Article  Google Scholar 

  35. Watts DJ, Strogatz SH (1998) Collective dynamics of ‘small-world’ networks. Nature 393:440–442

    Article  Google Scholar 

  36. Moffaert KV, Drugan MM, Ann N (2013) Scalarized multi-objective reinforcement learning: novel design techniques. In: 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), Singapore, Singapore

  37. Qin Y, Wang H, Yi S, Li X, Zhai L (2020) Virtual machine placement based on multi-objective reinforcement learning. Appl Intell 50:2370–2383

    Article  Google Scholar 

  38. Li X, Wang H, Yi S, Zhai L (2019) Cost-efficient disaster backup for multiple data centers using capacity-constrained multicast. Concurr Comput Pract Exp 31:1–18

    Article  Google Scholar 

  39. Naldi M (2005) Connectivity of Waxman topology models. Comput Commun 29(1):24–31

    Article  Google Scholar 

  40. Xu D, Chiang M, Rexford J (2011) Link-state routing with hop-by-hop forwarding can achieve optimal traffic engineering. IEEE/ACM Trans Netw 19(6):1717–1730

    Article  Google Scholar 

  41. Jain S, Kumar A, Mandal S, Ong J, Poutievski L, Singh A, Venkata S, Wanderer J, Zhou J, Zhu M (2013) B4: experience with a globally-deployed software defined wan. In: Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM, pp 3–14

  42. Zitzler E, Thiele L, Laumanns M, Fonseca CM, Da Fonseca VG (2003) Performance assessment of multiobjective optimizers: an analysis and review. IEEE Trans Evolut Comput 7(2):117–132

    Article  Google Scholar 

  43. Tsitsiklis JN (1994) Asynchronous stochastic approximation and Q-learning. Mach Learn 16(3):185–202

    Article  Google Scholar 

  44. Melo FS (2001) Convergence of Q-learning: a simple proof. Institute Of Systems and Robotics, Tech. Rep pp 1–4

  45. Han D-K, Mulyana B, Stanković V, Cheng S (2023) A survey on deep reinforcement learning algorithms for robotic manipulation. Sensors (Basel, Switzerland) 23:1–35

    Google Scholar 

Download references

Acknowledgements

We would like to thank the editor and reviewers for their helpful suggestions to improve this paper.

Funding

This work is partially supported by the National Natural Science Foundation of China (NSFC No. 62236003, 61771230), the Shandong Provincial Natural Science Foundation of China (Grant No. ZR2023MF090, ZR2023MF062), and the Introduction and Cultivation Program for Young Innovative Talents of Universities in Shandong Province of China (Grant No. 2021QCYY003).

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, Linyi University, Linyi, 276000, Shandong, China

    Xiaole Li, Haitao Liu & Haifeng Wang

  2. School of Software, Shandong University, Jinan, 250000, Shandong, China

    Xiaole Li

Authors
  1. Xiaole Li
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Haitao Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Haifeng Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

Contributions

Xiaole Li wrote and modified the main manuscript text, and provided financial support. Haitao Liu wrote and modified the main manuscript text. Haifeng Wang prepared figures and tables, modified manuscript content. All authors reviewed the manuscript.

Corresponding authors

Correspondence to Haitao Liu or Haifeng Wang.

Ethics declarations

Conflict of interest

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, X., Liu, H. & Wang, H. Data transmission optimization in edge computing using multi-objective reinforcement learning. J Supercomput 80, 21179–21206 (2024). https://doi.org/10.1007/s11227-024-06213-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-024-06213-7

Keywords