Skip to main content

Advertisement

SpringerLink
Log in

Multi-queue scheduling of heterogeneous jobs in hybrid geo-distributed cloud environment

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

In hybrid geo-distributed clouds, there is a technique named cloud bursting in which applications are handled in the private cloud with less expenses and burst into public clouds when the resources of the private cloud run out. However, how to deploy heterogeneous jobs in heterogeneous hybrid cloud environment is still a challenge. In this paper, a multi-queue scheduling approach of heterogeneous jobs for cloud bursting is proposed. In the private cloud, jobs are classified into I/O-intensive and CPU-intensive jobs, and nodes are divided into main I/O and CPU resource pools. Jobs are dispatched to corresponding resource pools to reduce the job execution time in heterogeneous cloud environment. A genetic algorithm is applied to schedule jobs to optimal job queues, which can reduce the job waiting time. Then, the execution time of each task is predicted by BP neural network. Jobs with high priority will be allocated to resources with the earliest finish time in the private cloud according to the prediction results. If the private cloud cannot meet the demand of users, public clouds with minimal costs will be applied. Experiments show that our proposed algorithm can reduce the average job response time and improve the throughput of the private cloud. It also can reduce the average task waiting time, average task execution time and average task response time significantly. Moreover, the costs of the hybrid clouds are reduced.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Hwang K, Bai X, Shi Y, Li M, Chen W, Yong Wu (2016) Cloud performance modeling with benchmark evaluation of elastic scaling strategies. IEEE Trans Parallel Distrib Syst 27(1):130–143

    Article  Google Scholar 

  2. Farahabady MRH, Lee YC, Zomaya AY (2014) Pareto-optimal cloud bursting. IEEE Trans Parallel Distrib Syst 25(10):2670–2682

    Article  Google Scholar 

  3. Yuan H, Bi J, Tan W, Li BH (2017) Temporal task scheduling with constrained service delay for profit maximization in hybrid clouds. IEEE Trans Autom Sci Eng 14(1):337–348

    Article  Google Scholar 

  4. TaoBao. https://www.taobao.com/. Accessed 24 June 2017

  5. Amazon. https://www.amazon.com/. Accessed 24 June 2017

  6. Clemente-Castello FJ, Mayo R, Fernandez JC (2017) cost model and analysis of iterative Mapreduce applications for hybrid cloud bursting. In: Proceedings of 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID). IEEE, pp 858–864

  7. Lee YC, Lian B (2017) Cloud Bursting scheduler for cost efficiency. In: Proceedings of 10th IEEE International Conference on Cloud Computing (CLOUD). IEEE, pp 774–777

  8. Nicolae B, Rafique MM, Mayo R (2017) Evaluation of data locality strategies for hybrid cloud bursting of iterative Mapreduce. In: Proceedings of 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID). IEEE, pp 181–185

  9. Guo T, Sharma U, Shenoy P, Wood T, Sahu S (2014) Cost-aware cloud bursting for enterprise applications. ACM Trans Internet Technol 13(3):10

    Article  Google Scholar 

  10. Lu P, Sun Q, Wu K, Zhu Z (2015) Distributed online hybrid cloud management for profit-driven multimedia cloud computing. IEEE Trans Multimed 17(8):1297–1308

    Article  Google Scholar 

  11. Charrada F, Tata S (2016) An efficient algorithm for the bursting of service-based applications in hybrid Clouds. IEEE Trans Serv Comput 9(3):357–367

    Article  Google Scholar 

  12. Malawski M, Figiela K, Nabrzyski J (2013) Cost minimization for computational applications on hybrid cloud infrastructures. Future Gener Comput Syst 29(7):1786–1794

    Article  Google Scholar 

  13. Kailasam S, Gnanasambandam N, Dharanipragada J, Sharma N (2013) Optimizing ordered throughput using autonomic cloud bursting schedulers. IEEE Trans Softw Eng 39(11):1564–1581

    Article  Google Scholar 

  14. Toosi AN, Sinnott R, Buyya R (2018) Resource provisioning for data-intensive applications with deadline constraints on hybrid clouds using Aneka. Future Gener Comput Syst 79(2):765–775

    Article  Google Scholar 

  15. Van den Bossche R, Vanmechelen K, Broeckhove J (2013) Online cost-efficient scheduling of deadline-constrained workloads on hybrid clouds. Future Gener Comput Syst 29(4):973–985

    Article  Google Scholar 

  16. Zhu J, Li X, Ruiz R, Xu X, Zhang Y (2016) Scheduling stochastic multi-stage jobs on elastic computing services in hybrid clouds. In: Proceedings of 23rd IEEE International Conference on Web Services (ICWS). IEEE, pp 678–681

  17. Genez TAL, Bittencourt L, Fonseca N, Madeira E (2015) Estimation of the available bandwidth in inter-cloud links for task scheduling in hybrid clouds. IEEE Trans Cloud Comput. https://doi.org/10.1109/TCC.2015.2469650

    Article  Google Scholar 

  18. Pelaez V, Campos A, Garcia DF, Entrialgo J (2016) Autonomic scheduling of deadline-constrained bag of tasks in hybrid clouds. In: Proceedings of International Symposium on Performance Evaluation of Computer and Telecommunication Systems (SPECTS). IEEE. https://doi.org/10.1109/spects.2016.7570526

  19. Champati JP, Liang B (2015) One-restart algorithm for scheduling and offloading in a hybrid cloud. In: Proceedings of 23rd IEEE International Symposium on Quality of Service (IWQoS). IEEE, pp 31–40

  20. Duan R, Prodan R, Li X (2014) Multi-objective game theoretic scheduling of bag-of-tasks workflows on hybrid clouds. IEEE Trans Cloud Comput 2(1):29–42

    Article  Google Scholar 

  21. Kliazovich D, Pecero JE, Tchernykh A, Bouvry P, Khan SU, Zomaya AY (2016) CA-DAG: modeling communication-aware applications for scheduling in cloud computing. J Grid Comput 14(1):23–39

    Article  Google Scholar 

  22. Singh S, Chana I (2015) QRSF: QoS-aware resource scheduling framework in cloud computing. J supercomput 71(1):241–292

    Article  Google Scholar 

  23. Wang X, Wang Y, Hao Z, Du J (2016) the research on resource scheduling based on fuzzy clustering in cloud computing. In: Proceedings of 8th International Conference on Intelligent Computation Technology and Automation (ICICTA). IEEE, pp 1025–1028

  24. Spicuglia S, Chen LY, Serazzi G, Binder W, Smirni E (2013) On load balancing: a mix-aware algorithm for heterogeneous systems. In: Proceedings of 4th ACM/SPEC International Conference on Performance Engineering (ICPE). ACM, pp 71–76

  25. Stanford large network dataset collection (SNAP). http://snap.stanford.edu/data/index.html. Accessed 16 July 2017

  26. Rasooli A, Down DG (2014) COSHH: a classification and optimization based scheduler for heterogeneous Hadoop systems. Future Gener Comput Syst 36(3):1–15

    Article  Google Scholar 

  27. Wang WJ, Chang YS, Lo WT, Lee YK (2013) Adaptive scheduling for parallel tasks with QoS satisfaction for hybrid cloud environments. J supercomput 66(2):783–811

    Article  Google Scholar 

  28. C. Software, SourceMonitor Version 3.4. http://www.campwoodsw.com/sourcemonitor.html. Accessed 2 June 2016

  29. Teng F, Yu L, Li T (2014) A novel real-time scheduling algorithm and performance analysis of a MapReduce-based cloud. J supercomput 69(2):739–765

    Article  Google Scholar 

Download references

Acknowledgements

The work was supported by the National Natural Science Foundation of China (NSFC) under Grants (Nos. 61472294, 61672397), Key Lab of Guangdong for Utilization of Remote Sensing and Geographical Information System, Guangzhou Institute of Geography (Grant No. 2017B030314138), Fundamental Research Funds for the Central Universities (WUT No.2017-YB-029), State Key Laboratory of Software Development Environment, Beihang University, (Grant No. SKLSDE-2017KF-04). Any opinions, findings, and conclusions are those of the authors and do not necessarily reflect the views of the above agencies.

Author information

Authors and Affiliations

  1. State Key Laboratory of Software Development Environment, Beihang University, Beijing, 100191, People’s Republic of China

    Li Chunlin

  2. School of Computer Science and Technology, Wuhan University of Technology, Wuhan, 430063, People’s Republic of China

    Li Chunlin, Tang Jianhang & Luo Youlong

  3. Key Lab of Guangdong for Utilization of Remote Sensing and Geographical Information System, Guangzhou Institute of Geography, Guangzhou, People’s Republic of China

    Li Chunlin

Authors
  1. Li Chunlin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tang Jianhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Luo Youlong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Li Chunlin.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chunlin, L., Jianhang, T. & Youlong, L. Multi-queue scheduling of heterogeneous jobs in hybrid geo-distributed cloud environment. J Supercomput 74, 5263–5292 (2018). https://doi.org/10.1007/s11227-018-2420-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-018-2420-8

Keywords

Search

Navigation