Skip to main content
SpringerLink
Log in

A heuristic multi-objective task scheduling framework for container-based clouds via actor-critic reinforcement learning

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Container-based cloud technology has changed the delivery mode of traditional applications and brought a breakthrough development to the field of cloud computing. However, the uncertainty of cloud environment and variability of application requirements increase the scheduling cost of tasks in container cloud. In particular, how to balance the business performance and utilization efficiency of cloud resources in the peak stage of application access is the focus of future for container cluster technology. In this paper, we propose a heuristics multi-objective task scheduling framework based on reinforcement learning (AC-CCTS). The proposed framework not only solves the problems of single objective and local convergence in traditional task scheduling methods, but also reduces the cost of experiential learning with reinforcement learning methods. Firstly, we define container cloud environment, scheduling agent, scheduling actions and scheduling evaluation methods to establish a deep reinforcement learning-based dynamic scheduling model. Then, based on Actor-Critic algorithm, we design heuristic rules and prioritized experience replay method to speed up convergence of task scheduling and decrease learning costs. At the same time, we provide compensation mechanism for dynamic task scheduling to improve the robustness of the approach. Finally, we implement comparative experiments to simulate various scheduling scenarios and verify the effectiveness of AC-CCTS from different perspectives such as resource balance, resource utilization and QoS. Compared with traditional meta-heuristic scheduling methods such as FIMPSO, HWOA-MBA and other reinforcement learning algorithms such as DeepRM-Plus and RLSched, AC-CCTS shows better resource utilization efficiency and convergence stability in container-based cloud task scheduling.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. We regard the task in our work as the represents of all types of cloud computing workloads about operation mode (e.g., long-running services, batch jobs, periodic jobs, etc.)

References

  1. Xie XL, Wang Q (2020) A scheduling algorithm based on multi-objective container cloud task. J Shandong Univ (Eng Sci) 50(04):14–21

    Google Scholar 

  2. Bian JF (2017) Docker-based application container cluster management system. Shandong University, Jinan

    Google Scholar 

  3. He Z (2020) Novel container cloud elastic scaling strategy based on Kubernetes. In: 2020 IEEE 5th information technology and mechatronics engineering conference (ITOEC), pp 1400–1404. https://doi.org/10.1109/ITOEC49072.2020.9141552

  4. Pahl C, Brogi A, Soldani J (2019) Cloud container technologies: a state-of-the-art review. IEEE Trans Cloud Comput 7(3):677–692. https://doi.org/10.1109/TCC.2017.2702586

    Article  Google Scholar 

  5. Avinab M, Wang Y, Zhang F et al (2018) Energy-aware fault-tolerant dynamic task scheduling scheme for virtualized cloud data centers. Mobile Netw Appl. https://doi.org/10.1007/s11036-018-1062-7

    Article  Google Scholar 

  6. Alsadie D (2021) A metaheuristic framework for dynamic virtual machine allocation with optimized task scheduling in cloud data centers. IEEE Access 9:74218–74233. https://doi.org/10.1109/ACCESS.2021.3077901

    Article  Google Scholar 

  7. Ebadifard F, Babamir SM (2021) Autonomic task scheduling algorithm for dynamic workloads through a load balancing technique for the cloud-computing environment. Clust Comput 24(2):1075–1101. https://doi.org/10.1007/s10586-020-03177-0

    Article  Google Scholar 

  8. Silver D, Huang A, Maddison CJ et al (2016) Mastering the game of go with deep neural networks and tree search. Nature 529(7587):484–489. https://doi.org/10.1038/nature16961

    Article  Google Scholar 

  9. Sutton RS, Barto AG (1998) Reinforcement learning. Bradford Book 15(7):665–685. https://doi.org/10.1007/978-3-642-27645-3

    Article  Google Scholar 

  10. Li H, Kumar N, Chen R, et al (2018) Deep reinforcement learning. In: ICASSP 2018 - 2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)

  11. Wang Z, Liu H, Han L et al (2021) Research and implementation of scheduling strategy in kubernetes for computer science laboratory in universities. Information (Switzerland) 12(1):16–26. https://doi.org/10.3390/info12010016

    Article  Google Scholar 

  12. Rossi F, Cardellini V, Presti F L (2019) Elastic deployment of software containers in geo-distributed computing environments. In: 2019 IEEE symposium on computers and communications (ISCC). https://doi.org/10.1109/ISCC47284.2019.8969607

  13. Menouer T (2021) KCSS: Kubernetes container scheduling strategy. J Supercomput 77(5):4267–4293. https://doi.org/10.1007/s11227-020-03427-3

    Article  Google Scholar 

  14. Jaybhaye S M, Attar V Z (2021) Heterogeneous resource provisioning for workflow-based applications using AHP in cloud computing. In: Proceedings of international conference on communication and computational technologies. Springer, Singapore, pp 453–465. https://doi.org/10.1007/978-981-15-5077 5_41

  15. Zhou Z, Xie H, Li F (2019) A novel task scheduling algorithm integrated with priority and greedy strategy in cloud computing. J Intell Fuzzy Syst 37(4):1–9. https://doi.org/10.3233/JIFS-179299

    Article  Google Scholar 

  16. Zhou Z, Wang H, Shao H et al (2020) A high-performance scheduling algorithm using greedy strategy toward quality of service in the cloud environments. Peer-to-Peer Netw Appl 13(6):2214–2223. https://doi.org/10.1007/s12083020-00888-4

    Article  Google Scholar 

  17. Kumar S, Dumka A (2021) Load balancing with the help of round robin and shortest job first scheduling algorithm in cloud computing. In: Proceedings of international conference on machine intelligence and data science applications. Springer, Singapore, pp 213–223. https://doi.org/10.1007/978-981-33-4087-9_19

  18. Gao Y (2021) Min-max scheduling of batch or drop-line jobs under agreeable release and processing times. Asia-Pacific J Oper Res. https://doi.org/10.1142/S0217595921500238

    Article  MATH  Google Scholar 

  19. Singh S, Singh V (2016) A genetic based improved load balanced Min-Min task scheduling algorithm for load balancing in cloud computing. In: 2016 8th international conference on computational intelligence and communication networks. IEEE, pp 677–681. https://doi.org/10.1109/CICN.2016.139

  20. Samadi Y, Zbakh M., Tadonki C (2018) E-HEFT: Enhancement heterogeneous earliest finish time algorithm for task scheduling based on load balancing in cloud computing. In: 2018 international conference on high performance computing and simulation (HPCS), pp 601–609. https://doi.org/10.1109/HPCS.2018.00100

  21. Soltani N, Barekatain B, Neysiani BS (2021) MTC: Minimizing time and cost of cloud task scheduling based on customers and providers needs using genetic algorithm. Int J Intell Syst Appl 13(2):38–51. https://doi.org/10.5815/ijisa.2021.02.03

    Article  Google Scholar 

  22. Peng L, Song J, He X et al (2018) Resource scheduling optimisation algorithm for containerised microservice architecture in cloud computing. Int J High Perform Syst Archit 8(1–2):51–58. https://doi.org/10.1504/IJHPSA.2018.094144

    Article  Google Scholar 

  23. Devaraj AFS, Elhoseny M, Dhanasekaran S et al (2020) Hybridization of firefly and improved multi-objective particle swarm optimization algorithm for energy efficient load balancing in cloud computing environments. J Parallel Distrib Comput 142:36–45. https://doi.org/10.1016/j.jpdc.2020.03.022

    Article  Google Scholar 

  24. Li H, Wang X, Gao S, et al (2020) A service performance aware scheduling approach in containerized cloud. In: 2020 IEEE 3rd international conference on computer and communication engineering technology (CCET). IEEE, pp 194–198. https://doi.org/10.1109/CCET50901.2020.9213084

  25. Manikandan N, Gobalakrishnan N, Pradeep K (2022) Bee optimization based random double adaptive whale optimization model for task scheduling in cloud computing environment. Comput Commun 187:35–44. https://doi.org/10.1016/j.comcom.2022.01.016

    Article  Google Scholar 

  26. Jian CF, Chen JW, Zhang MY (2019) Improved chaotic bat swarm cooperative scheduling algorithm for edge computing. J Chin Comput Syst 40(11):2424–2430

    Google Scholar 

  27. Chang J, Hu Z, Tao Y (2018) Task scheduling based on dynamic non-linear PSO in cloud environment. In: 2018 IEEE 9th international conference on software engineering and service science (ICSESS). Beijing, China, pp 877–880, https://doi.org/10.1109/ICSESS.2018.8663825

  28. Ziyath SPM, Senthilkumar S (2020) MHO: meta heuristic optimization applied task scheduling with load balancing technique for cloud infrastructure services. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-020-02282-7

    Article  Google Scholar 

  29. Ding S, Wu J, Xie G and Zeng G (2017) A hybrid heuristic-genetic algorithm with adaptive parameters for static task scheduling in heterogeneous computing system. In: 2017 IEEE Trustcom/BigDataSE/ICESS, pp 761–766. https://doi.org/10.1109/Trustcom/BigDataSE/ICESS.2017.310

  30. Zou J, Hao T, Yu C et al (2021) A3C-DO: A regional resource scheduling framework based on deep reinforcement learning in edge scenario. IEEE Trans Comput 70(2):228–239. https://doi.org/10.1109/TC.2020.2987567

    Article  MATH  Google Scholar 

  31. Mao H, Alizadeh M, Menache I, et al (2016) Resource management with deep reinforcement learning. In Proceedings of the 15th ACM workshop on tot topics in networks, pp 50–56. https://doi.org/10.1145/3005745.3005750

  32. Guo W, Tian W, Ye Y et al (2020) Cloud resource scheduling with deep reinforcement learning and imitation learning. IEEE Internet Things J 8(5):3576–3586. https://doi.org/10.1109/JIOT.2020.3025015

    Article  Google Scholar 

  33. Che H, Bai Z, Zuo R et al (2020) (2020) A deep reinforcement learning approach to the optimization of data center task scheduling. Complexity 4:1–12. https://doi.org/10.1155/2020/3046769

    Article  Google Scholar 

  34. Lorido-Botran T, Bhatti M K (2021) Adaptive container scheduling in cloud data centers: a deep reinforcement learning approach. In: International conference on advanced information networking and applications. Springer, Cham, pp 572–581. https://doi.org/10.1007/978-3-030-75078-7_57

  35. Asghari A, Sohrabi MK, Yaghmaee F (2020) Task scheduling, resource provisioning, and load balancing on scientific workflows using parallel SARSA reinforcement learning agents and genetic algorithm. J Supercomput 77(3):2800–2828. https://doi.org/10.1007/s11227-020-03364-1

    Article  Google Scholar 

  36. Pham N, Nguyen L, Phan D, et al (2020) A hybrid stochastic policy gradient algorithm for reinforcement learning. In: International conference on artificial intelligence and statistics. PMLR, pp 374–385

  37. Alibaba Inc (2018) Alibaba production cluster data v2018. Website. https://github.com/alibaba/-clusterdata/tree/v2018

Download references

Author information

Authors and Affiliations

  1. School of Information Science and Technology, University of Science and Technology of China, Hefei, 230026, China

    Lilu Zhu, Feng Wu & Xinmei Tian

  2. Aerospace Information Research Institute, Chinese Academy of Sciences, Suzhou, 215123, China

    Lilu Zhu & Kai Huang

  3. Aerospace Information Research Institute, Chinese Academy of Sciences, Beijing, 100094, China

    Yanfeng Hu

Authors
  1. Lilu Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Feng Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yanfeng Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kai Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xinmei Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lilu Zhu.

Ethics declarations

Conflict of interest

No potential conflict of interest was reported by the authors.

Data availability

The datasets analysed during this study are available in the github repository: https://github.com/alibaba/-clusterdata/tree/v2018.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, L., Wu, F., Hu, Y. et al. A heuristic multi-objective task scheduling framework for container-based clouds via actor-critic reinforcement learning. Neural Comput & Applic 35, 9687–9710 (2023). https://doi.org/10.1007/s00521-023-08208-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08208-6

Keywords

Search

Navigation