Skip to main content
Springer Nature Link
Log in

An energy-efficient big data workflow scheduling algorithm under budget constraints for heterogeneous cloud environment

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Infrastructure as a service model of cloud computing provides a tremendous amount of high-performance computing systems for the execution of scientific workflow applications. However, due to explosive growth in energy consumption and high charge cost of using these cloud systems, energy-efficient workflow scheduling under budget constraints becomes the most challenging issue. Very few research works have been done that consider the stated issue. Most of them mainly focus on the minimization of schedule length under user-specified budget constraints or energy consumption constraints. In this article, we propose an energy-efficient workflow scheduling algorithm named reducing energy consumption using fair pre-assignment of available budget (RECFPAB) that reduces energy consumption under client-specified budget constraints. The RECFPAB introduces a flexible mechanism to save energy consumption with the inclusion of energy and cost coefficient factor that enables fair distribution of available budget for unscheduled tasks of the workflow application. In order to compare the performance of the proposed algorithm, an energy-efficient version of the popular existing algorithms such as heterogeneous budget constrained scheduling and minimizing schedule length using budget level are introduced. The experimental evaluation based on Genome, LIGO, and Montage applications shows that RECFPAB gives significant results in comparison with considered algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Keahey K, Raicu I, Chard K, Nicolae B (2016) Guest editors introduction: SPECIAL issue on scientific cloud computing. IEEE Trans Cloud Comput 4(1):4–5

    Google Scholar 

  2. Li H, Ota K, Dong M, Vasilakos AV, Nagano K (2020) Multimedia processing pricing strategy in GPU-accelerated cloud computing. IEEE Trans Cloud Comput 8(4):1264–1273. https://doi.org/10.1109/TCC.2017.2672554

    Article  Google Scholar 

  3. Kumrai T, Ota K, Dong M, Kishigami J, Sung DK (2016) Multiobjective optimization in cloud brokering systems for connected internet of things. IEEE Internet Things J 4(2):404–413

    Google Scholar 

  4. Rodriguez MA, Buyya R (2014) Deadline based resource provisioning and scheduling algorithm for scientific workflows on clouds. IEEE Trans Cloud Comput 2(2):222–235

    Google Scholar 

  5. Kintsakis AM, Psomopoulos FE, Mitkas PA (2019) Reinforcement learning based scheduling in a workflow management system. Eng Appl Artif Intell 81:94–106

    Google Scholar 

  6. Andrae ASG, Edler T (2015) On global electricity usage of communication technology: trends to 2030. Challenges 6(1):117–157

    Google Scholar 

  7. Ismayilov G, Topcuoglu HR (2020) Neural network based multi-objective evolutionary algorithm for dynamic workflow scheduling in cloud computing. Futur Gener Comput Syst 102:307–322

    Google Scholar 

  8. Verma A, Kaushal S (2017) A hybrid multi-objective particle swarm optimization for scientific workflow scheduling. Parallel Comput 62:1–19

    MathSciNet  Google Scholar 

  9. Chen J, Li K, Deng Q, Li K, Yu PS (2019) Distributed deep learning model for intelligent video surveillance systems with edge computing. IEEE Trans Ind Inform. https://doi.org/10.1109/TII.2019.2909473

    Article  Google Scholar 

  10. Ge R, Feng X, Cameron KW (2005) Performance-constrained distributed dvs scheduling for scientific applications on power-aware clusters. In: SC'05: 2005 ACM/IEEE conference on supercomputing. pp 34–34

  11. Xiao X, Xie G, Li R, Li K (2016) Minimizing schedule length of energy consumption constrained parallel applications on heterogeneous distributed systems. In: 2016 IEEE Trustcom/BigDataSE/ISPA. pp 1471–1476

  12. Song J, Xie G, Li R, Chen X (2017) An efficient scheduling algorithm for energy consumption constrained parallel applications on heterogeneous distributed systems. In: 2017 IEEE international symposium on parallel and distributed processing with applications and 2017 IEEE international conference on ubiquitous computing and communications (ISPA/IUCC). pp 32–39

  13. Li J, Xie G, Li K, Tang Z (2019) Enhanced parallel application scheduling algorithm with energy consumption constraint in heterogeneous distributed systems. J Circuit Syst Comput 28(11):1950190

    Google Scholar 

  14. Bunde DP (2009) Power-aware scheduling for makespan and flow. J Shed 12(5):489–500

    MathSciNet  MATH  Google Scholar 

  15. Qin Y, Wang H, Yi S, Li X, Zhai L (2020) An energy-aware scheduling algorithm for budget-constrained scientific workflows based on multi- objective reinforcement learning. J Supercomput 76(1):455–480

    Google Scholar 

  16. Hu Y, Li J, He L (2020) A reformed task scheduling algorithm for heterogeneous distributed systems with energy consumption constraints. Neural Comput Appl 32(10):5681–5693

    Google Scholar 

  17. Saharawat S, Kalra M (2020) Deadline constrained energy-efficient workflow scheduling heuristic for cloud. In: 2019 international conference on iot inclusive life (ICIIL 2019), NITTTR Chandigarh, India. pp 365–382

  18. Qureshi B (2019) Profile-based power-aware workflow scheduling framework for energy-efficient data centers. Future Gener Comput Syst 1(94):453–467

    Google Scholar 

  19. Singh V, Gupta I, Jana PK (2019) An energy efficient algorithm for workflow scheduling in IAAS cloud. J Grid Comput 3:1–20

    Google Scholar 

  20. Ahmad W, Alam S, Ahuja S, Malik S (2020) A dynamic VM provisioning and de-provisioning based cost-efficient deadline-aware scheduling algorithm for Big Data workflow applications in a cloud environment. Cluster Comput. 24:1–30

    Google Scholar 

  21. Chen W, Xie G, Li R, Bai Y, Fan C, Li K (2017) Efficient task scheduling for budget constrained parallel applications on heterogeneous cloud computing systems. Future Gener Comput Syst 74:1–1

    Google Scholar 

  22. Arabnejad H, Barbosa JG (2014) A budget constrained scheduling algorithm for workflow applications. J Grid Comput 12(4):665–679

    Google Scholar 

  23. Amazon EC2 Pricing. https://aws.amazon.com/ec2/pricing/. Accessed 13 Sept 2020

  24. Wei T, Zhou J, Cao K, Cong P, Chen M, Zhang G, Hu XS, Yan J (2017) Cost-constrained QoS optimization for approximate computation real-time tasks in heterogeneous MPSoCs. IEEE T Comput Aid D 37(9):1733–1746

    Google Scholar 

  25. Wang S, Qian Z, Yuan J, You I (2017) A DVFS based energy-efficient tasks scheduling in a data center. IEEE Access 11(5):13090–13102

    Google Scholar 

  26. Buyya R, Yeo CS, Venugopal S, Broberg J, Brandic I (2009) Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility. Future Gener Comput Syst 25(6):599–616

    Google Scholar 

  27. Katarya R, Arora Y (2020) Capsmf: a novel product recommender system using deep learning based text analysis model. Multimed Tools Appl 79(47):35927–35948

    Google Scholar 

  28. Mishra A, Gupta N, Gupta BB (2021) Defense mechanisms against DDoS attack based on entropy in SDN-cloud using POX controller. In: Telecommunication systems. pp 1–16

  29. Gupta BB (2020) An efficient KP design framework of attribute-based searchable encryption for user level revocation in cloud. Concurr Comput Pract Exp 32(18):e5291

    Google Scholar 

  30. Sahni J, Vidyarthi DP (2015) A cost-effective deadline-constrained dynamic scheduling algorithm for scientific workflows in a cloud environment. IEEE Trans Cloud Comput 6(1):2–18

    Google Scholar 

  31. Ahmad W, Alam B, Malik S (2019) Performance analysis of list scheduling algorithms by random synthetic DAGs. In: 2019 2nd international conference on advanced computing and software engineering (ICACSE)

  32. Altmann J, Kashef MM (2014) Cost model based service placement in federated hybrid clouds. Future Gener Comput Syst 1(41):79–90

    Google Scholar 

  33. McGough AS, Forshaw M, Gerrard C, Wheater S, Allen B, Robinson P (2014) Comparison of a cost-effective virtual cloud cluster with an existing campus cluster. Future Gener Comput Syst 1(41):65–78

    Google Scholar 

  34. Xie G, Chen Y, Xiao X, Xu C, Li R, Li K (2017) Energy-efficient fault-tolerant scheduling of reliable parallel applications on heterogeneous distributed embedded systems. IEEE Trans Sustain Energy 3(3):167–181

    Google Scholar 

  35. Li K, Tang X, Li K (2013) Energy-efficient stochastic task scheduling on heterogeneous computing systems. IEEE Trans Parallel Distrib Syst 25(11):2867–2876

    Google Scholar 

  36. Wang Y, Li K, Chen H, He L, Li K (2014) Energy-aware data allocation and task scheduling on heterogeneous multiprocessor systems with time constraints. IEEE Trans Emerg Topics Comput 2(2):134–148

    Google Scholar 

  37. Xiao X, Xie G, Li R, Li K (2016) Minimizing schedule length of energy consumption constrained parallel applications on heterogeneous distributed systems In: 2016 IEEE Trustcom/BigDataSE/ISPA 2016. pp 1471–1476

  38. Lee YC, Zomaya AY (2010) Energy conscious scheduling for distributed computing systems under different operating conditions. IEEE Trans Parallel Distrib Syst 22(8):1374–1381

    Google Scholar 

  39. Huang Q, Su S, Li J, Xu P, Shuang K, Huang X (2012) Enhanced energy-efficient scheduling for parallel applications in cloud. In: 2012 12th IEEE/ACM international symposium on cluster, cloud and grid computing (ccgrid 2012). pp 781–786

  40. Xie G, Zeng G, Li R, Li K (2017) Energy-aware processor merging algorithms for deadline constrained parallel applications in heterogeneous cloud computing. IEEE Trans Sustain Energ 2(2):62–75

    Google Scholar 

  41. Sun D, Zhang G, Yang S, Zheng W, Khan SU, Li K (2015) Re-stream: real-time and energy-efficient resource scheduling in big data stream computing environments. Inform Sci 319:92–112

    MathSciNet  Google Scholar 

  42. Durillo JJ, Nae V, Prodan R (2014) Multi-objective energy efficient workflow scheduling using list-based heuristics. Futur Gener Comput Syst 36:221–236

    Google Scholar 

  43. Zong Z, Manzanares A, Ruan X, Qin X (2011) EAD and PEBD: two energy-aware duplication scheduling algorithms for parallel tasks on homogeneous clusters. IEEE Trans Comput 60(3):360–374

    MathSciNet  MATH  Google Scholar 

  44. Xie G, Jiang J, Liu Y, Li R, Li K (2017) Minimizing energy consumption of real-time parallel applications using downward and upward approaches on heterogeneous systems. IEEE Trans Ind Inf 13(3):1068–1078

    Google Scholar 

  45. Xie G, Zeng G, Xiao X, Li R, Li K (2017) Energy efficient scheduling algorithms for real-time parallel applications on heterogeneous distributed embedded systems. IEEE Trans Parallel Distrib Syst 28(12):3426–3442

    Google Scholar 

  46. Zhou AC, He B, Liu C (2015) Monetary cost optimizations for hosting workflow-as-a-service in IaaS clouds. IEEE Trans Cloud Comput 4(1):34–48

    Google Scholar 

  47. Mao M, Humphrey M (2011) Auto-scaling to minimize cost and meet application deadlines in cloud workflows. In: 2011 international conference for high performance computing, networking, storage and analysis. pp 1–-12

  48. Yu J, Buyya R, Tham CK (2005) QoS-based scheduling of workflow applications on service grids. In: 2005 1st IEEE international conference on e-science and grid computing. e-Science 2005, IEEE CS Press, Los Alamitos, pp 5–8

  49. Yuan Y, Li X, Wang Q, Zhu X (2009) Deadline division-based heuristic for cost optimization in workflow scheduling. Inf Sci 179(15):2562–2575

    MATH  Google Scholar 

  50. Wu CQ, Lin X, Yu D, Xu W, Li L (2014) End-to-end delay minimization for scientific workflows in clouds under budget constraint. IEEE Trans Cloud Comput 3(2):169–181

    Google Scholar 

  51. Al-Qerem A, Alauthman M, Almomani A, Gupta BB (2020) IoT transaction processing through cooperative concurrency control on fog–cloud computing environment. Soft Comput 24(8):5695–5711

    Google Scholar 

  52. Gupta BB, Quamara M (2020) An overview of internet of things (IoT): Architectural aspects, challenges, and protocols. Concurr Comput Pract Exp 32(21):e4946

    Google Scholar 

  53. Katarya R, Meena SK (2020) Machine learning techniques for heart disease prediction: a comparative study and analysis. Health Technol 1–11

  54. Gupta A, Katarya R (2020) Social media based surveillance systems for healthcare using machine learning: a systematic review. J Biomed Inf 103500

  55. Amazon Web Services (AWS). https://aws.amazon.com. Accessed 20 Sept 2020

  56. Ullman JD (1975) NP-complete scheduling problems J. Comput Sys Sci 10(3):384–393

    MathSciNet  MATH  Google Scholar 

  57. Chen Y, Xie G, Li R (2018) Reducing energy consumption with cost budget using available budget pre-assignment in heterogeneous cloud computing systems. IEEE Access 11(6):20572–20583

    Google Scholar 

  58. Scientific workflow applications. https://confluence.pegasus.isi.edu/display/pegasus/WorkflowHub. Accessed 6 Sept 2020

  59. Li H, Ruan J, Durbin R (2008) Mapping short DNA sequencing reads and calling variants using mapping quality scores. Genome Res 18(11):1851–1858

    Google Scholar 

  60. Abbott BP, Abbott R, Adhikari R, Ajith P, Allen B, Allen G, Amin RS, Anderson SB, Anderson WG, Arain MA, Araya M (2009) LIGO: the laser interferometer gravitational-wave observatory. Rep Prog Phys 72(7):076901

    Google Scholar 

  61. Berriman GB, Deelman E, Good JC, Jacob JC, Katz DS, Kesselman C, Laity AC, Prince TA, Singh G, Su MH (2004) Montage: a grid-enabled engine for delivering custom science-grade mosaics on demand. In: Optimizing scientific return for astronomy through information technologies, vol 5493, pp. 221–232

  62. Juve G, Chervenak A, Deelman E, Bharathi S, Mehta G, Vahi K (2013) Characterizing and profiling scientific workflows. Future Gener Comput Syst 29(3):682–692

    Google Scholar 

  63. Bharathi S, Chervenak A, Deelman E, Mehta G, Su MH, Vahi K (2008) Characterization of scientific workflows. In: 2008 3rd workshop workflows support large-scale science. pp 1–10

Download references

Acknowledgements

This work is supported by Visvesvaraya PhD Scheme, MeitY, Govt. of India. [MEITY-PHD-1246].

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Faculty of Engineering and Technology, Jamia Millia Islamia, New Delhi, 110025, India

    Wakar Ahmad & Bashir Alam

  2. International Institute of Information Technology, Hyderabad, 500032, India

    Aman Atman

Authors
  1. Wakar Ahmad
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Bashir Alam
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Aman Atman
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Wakar Ahmad.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ahmad, W., Alam, B. & Atman, A. An energy-efficient big data workflow scheduling algorithm under budget constraints for heterogeneous cloud environment. J Supercomput 77, 11946–11985 (2021). https://doi.org/10.1007/s11227-021-03733-4

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-021-03733-4

Keywords