Skip to main content
SpringerLink
Log in

Intelligent workflow scheduling for Big Data applications in IoT cloud computing environments

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

Effective task scheduling is recognized as one of the main critical challenges in cloud computing; it is an essential step for effectively exploiting cloud computing resources, as several tasks may need to be efficiently scheduled on various virtual machines by minimizing makespan and maximizing resource utilization. Task scheduling is an NP-hard problem, and consequently, finding the best solution may be difficult, particularly for Big Data applications. This paper presents an intelligent Big Data task scheduling approach for IoT cloud computing applications using a hybrid Dragonfly Algorithm. The Dragonfly algorithm is a newly introduced optimization algorithm for solving optimization problems which mimics the swarming behaviors of dragonflies. Our algorithm, MHDA, aims to decrease the makespan and increase resource utilization, and is thus a multi-objective approach. β-hill climbing is utilized as a local exploratory search to enhance the Dragonfly Algorithm’s exploitation ability and avoid being trapped in local optima. Two experimental studies were conducted on synthetic and real trace datasets using the CloudSim toolkit to compare MHDA to other well-known algorithms for solving task scheduling problems. The analysis, which included the use of a t-test, revealed that MHDA outperformed other well-known algorithms: MHDA converged faster than other methods, making it useful for Big Data task scheduling applications, and it achieved 17.12% improvement in the results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Thanka, M.R., Maheswari, P.U., Edwin, E.B.: A hybrid algorithm for efficient task scheduling in cloud computing environment. Int. J. Reason. Based Intell. Syst. 11, 134–140 (2019)

    Google Scholar 

  2. Kumar, M., Sharma, S., Goel, A., Singh, S.: A comprehensive survey for scheduling techniques in cloud computing. J. Netw. Comput. Appl. 143(1), 1–33 (2019)

    Google Scholar 

  3. Lin, W., Peng, G., Bian, X., Xu, S., Chang, V., Li, Y.: Scheduling algorithms for heterogeneous cloud environment: main resource load balancing algorithm and time balancing algorithm. J. Grid Comput. 17, 699–726 (2019)

    Google Scholar 

  4. Xu, X., Fu, S., Li, W., Dai, F., Gao, H., Chang, V.: Multi-objective data placement for workflow management in cloud infrastructure using NSGA-II. IEEE Trans. Emerg. Top. Comput. Intell. 4, 605–615 (2020)

    Google Scholar 

  5. Zhu, Q.: Research on road traffic situation awareness system based on image big data. IEEE Intell. Syst. 35, 18–26 (2019)

    Google Scholar 

  6. Abdullahi, M., Ngadi, M.A., Dishing, S.I., Ahmad, B.I., et al.: An efficient symbiotic organisms search algorithm with chaotic optimization strategy for multi-objective task scheduling problems in cloud computing environment. J. Netw. Comput. Appl. 133, 60–74 (2019)

    Google Scholar 

  7. Mohammadi, A., Rezvani, M.H.: A novel optimized approach for resource reservation in cloud computing using producer-consumer theory of microeconomics. J. Supercomput. 75, 7391–7425 (2019)

    Google Scholar 

  8. Geng, X., Yu, L., Bao, J., Fu, G.: A task scheduling algorithm based on priority list and task duplication in cloud computing environment. In: Web Intelligence, vol. 17. IOS Press, Amsterdam, pp. 121–129 (2019)

  9. Beegom, A.A., Rajasree, M.: Integer-PSO: a discrete PSO algorithm for task scheduling in cloud computing systems. Evol. Intell. 12, 227–239 (2019)

    Google Scholar 

  10. Cao, B., Wang, X., Zhang, W., Song, H., Lv, Z.: A many-objective optimization model of industrial Internet of Things based on private blockchain. IEEE Netw. 34, 78–83 (2020)

    Google Scholar 

  11. Singh, S., Jeong, Y.-S., Park, J.H.: A survey on cloud computing security: issues, threats, and solutions. J. Netw. Comput. Appl. 75, 200–222 (2016)

    Google Scholar 

  12. Singh, A., Chatterjee, K.: Cloud security issues and challenges: a survey. J. Netw. Comput. Appl. 79, 88–115 (2017)

    Google Scholar 

  13. Liu, B., Li, P., Lin, W., Shu, N., Li, Y., Chang, V.: A new container scheduling algorithm based on multi-objective optimization. Soft Comput. 22, 7741–7752 (2018)

    Google Scholar 

  14. Matos, J.G.D., Marques, C.K.D.M., Liberalino, C.H.: Genetic and static algorithm for task scheduling in cloud computing. Int. J. Cloud Comput. 8, 1–19 (2019)

    Google Scholar 

  15. Li, Y., Wang, S., Hong, X., Li, Y., Multi-objective task scheduling optimization in cloud computing based on genetic algorithm and differential evolution algorithm. In: 37th Chinese Control Conference (CCC). IEEE 2018, pp. 4489–4494 (2018)

  16. Alresheedi, S.S., Lu, S., Elaziz, M.A., Ewees, A.A.: Improved multiobjective salp swarm optimization for virtual machine placement in cloud computing. Hum. Cent. Comput. Inf. Sci. 9, 15 (2019)

    Google Scholar 

  17. Gawali, M.B., Shinde, S.K.: Standard deviation based modified cuckoo optimization algorithm for task scheduling to efficient resource allocation in cloud computing. J. Adv. Inf. Technol. (2017). https://doi.org/10.12720/jait.8.4.210-218

    Article  Google Scholar 

  18. Sundarrajan, R., Vasudevan, V.: An optimization algorithm for task scheduling in cloud computing based on multi-purpose cuckoo seek algorithm. In: International Conference on Theoretical Computer Science and Discrete Mathematics. Springer, Cham, pp. 415–424 (2016)

  19. Wang, G.-G.: Moth search algorithm: a bio-inspired metaheuristic algorithm for global optimization problems. Memetic Comput. 10, 151–164 (2018)

    Google Scholar 

  20. Elaziz, M.A., Xiong, S., Jayasena, K., Li, L.: Task scheduling in cloud computing based on hybrid moth search algorithm and differential evolution. Knowl. Based Syst. 169, 39–52 (2019)

    Google Scholar 

  21. Dhiman, G., Singh, K.K., Slowik, A., Chang, V., Yildiz, A.R., Kaur, A., Garg, M.: EMoSOA: a new evolutionary multi-objective seagull optimization algorithm for global optimization. Int. J. Mach. Learn. Cybern. 12, 571–596 (2021)

    Google Scholar 

  22. Mapetu, J.P.B., Chen, Z., Kong, L.: Low-time complexity and low-cost binary particle swarm optimization algorithm for task scheduling and load balancing in cloud computing. Appl. Intell. 49(3), 1–23 (2019)

    Google Scholar 

  23. Zhang, L., Liu, L., Yang, X.-S., Dai, Y.: A novel hybrid firefly algorithm for global optimization. PLoS ONE 11, e0163230 (2016)

    Google Scholar 

  24. Yang, X.-S.: Firefly algorithm, levy flights and global optimization. In: Research and Development in Intelligent Systems, vol. XXVI. Springer, London, pp. 209–218 (2010)

  25. Masadeh, R., Sharieh, A., Mahafzah, B.: Humpback whale optimization algorithm based on vocal behavior for task scheduling in cloud computing. Int. J. Adv. Sci. Technol. 13, 121–140 (2019)

    Google Scholar 

  26. Zheng, X.-L., Wang, L., A pareto based fruit fly optimization algorithm for task scheduling and resource allocation in cloud computing environment. In: IEEE Congress on Evolutionary Computation (CEC). IEEE 2016, pp. 3393–3400 (2016)

  27. Kashikolaei, S.M.G., Hosseinabadi, A.A.R., Saemi, B., Shareh, M.B., Sangaiah, A.K., Bian, G.-B.: An enhancement of task scheduling in cloud computing based on imperialist competitive algorithm and firefly algorithm. J. Supercomput. 76, 6302–6329 (2019)

    Google Scholar 

  28. Abualigah, L., Yousri, D., Abd Elaziz, M., Ewees, A.A., Al-ganess, M.A., Gandomi, A.H.: Aquila optimizer: a novel meta-heuristic optimization algorithm. Comput. Ind. Eng. (2021). https://doi.org/10.1016/j.cie.2021.107250

    Article  Google Scholar 

  29. Abualigah, L., Diabat, A., Mirjalili, S., Abd Elaziz, M., Gandomi, A.H.: The arithmetic optimization algorithm. Comput. Methods Appl. Mech. Eng. 376, 113609 (2021)

    MathSciNet  MATH  Google Scholar 

  30. Abdullahi, M., Ngadi, M.A., et al.: Symbiotic organism search optimization based task scheduling in cloud computing environment. Future Gen. Comput. Syst. 56, 640–650 (2016)

    Google Scholar 

  31. Abualigah, L., Diabat, A.: Advances in sine cosine algorithm: a comprehensive survey. Artif. Intell. Rev. (2021). https://doi.org/10.1007/s10462-020-09909-3

    Article  Google Scholar 

  32. Abualigah, L.M., Sawaie, A.M., Khader, A.T., Rashaideh, H., Al-Betar, M.A., Shehab, M.: \(\beta\)-hill climbing technique for the text document clustering. In: Proceedings of the New Trends in Information Technology (NTIT-2017). The University of Jordan, Amman, Jordan, 25–27 April 2017

  33. Abualigah, L.M., Khader, A.T., Al-Betar, M.A., Alyasseri, Z.A.A., Alomari, O.A., Hanandeh, E.S.: Feature selection with \(\beta\)-hill climbing search for text clustering application. In: 2017 Palestinian International Conference on Information and Communication Technology (PICICT), IEEE, pp. 22–27 (2017)

  34. Alyasseri, Z.A.A. , Khader, A.T., Al-Betar, M.A., Abualigah, L.M.: ECG signal denoising using \(\beta\)-hill climbing algorithm and wavelet transform. In: 2017 8th International Conference on Information Technology (ICIT), IEEE, pp. 96–101 (2017)

  35. Abualigah, L.: Multi-verse optimizer algorithm: a comprehensive survey of its results, variants, and applications. Neural Comput. Appl. 32(2), 1–21 (2020)

    Google Scholar 

  36. Abualigah, L., Diabat, A., Geem, Z.W.: A comprehensive survey of the harmony search algorithm in clustering applications. Appl. Sci. 10, 3827 (2020)

    Google Scholar 

  37. Alazzam, H., Alhenawi, E., Al-Sayyed, R.: A hybrid job scheduling algorithm based on tabu and harmony search algorithms. J. Supercomput. 75(12), 7994–8011 (2019)

    Google Scholar 

  38. Taherian Dehkordi, S., Khatibi Bardsiri, V.: Optimization task scheduling algorithm in cloud computing. J. Adv. Comput. Eng. Technol. 1, 17–22 (2015)

    Google Scholar 

  39. Kumar, K.P., Kousalya, K.: Amelioration of task scheduling in cloud computing using crow search algorithm. Neural Comput. Appl. 32, 5901–5907 (2019)

    Google Scholar 

  40. Agarwal, M., Srivastava, G. M. S.: A pso algorithm-based task scheduling in cloud computing. In: Soft Computing: Theories and Applications. Springer, Singapore, pp. 295–301 (2019)

  41. Abualigah, L., Diabat, A.: A novel hybrid antlion optimization algorithm for multi-objective task scheduling problems in cloud computing environments. Clust. Comput. 24, 205–223 (2020)

    Google Scholar 

  42. Mirjalili, S.: Dragonfly algorithm: a new meta-heuristic optimization technique for solving single-objective, discrete, and multi-objective problems. Neural Comput. Appl. 27, 1053–1073 (2016)

    Google Scholar 

  43. Al-Betar, M.A.: \(\beta\)-hill climbing: an exploratory local search. Neural Comput. Appl. 28, 153–168 (2017)

    Google Scholar 

  44. Ahmed, M., Mahmood, A.N., Hu, J.: A survey of network anomaly detection techniques. J. Netw. Comput. Appl. 60, 19–31 (2016)

    Google Scholar 

  45. Zhou, D., Yan, Z., Fu, Y., Yao, Z.: A survey on network data collection. J. Netw. Comput. Appl. 116, 9–23 (2018)

    Google Scholar 

  46. Valarmathi, R., Sheela, T.: Ranging and tuning based particle swarm optimization with bat algorithm for task scheduling in cloud computing. Clust. Comput. 22, 11975–11988 (2017)

    Google Scholar 

  47. Varshney, S., Singh, S.: An optimal bi-objective particle swarm optimization algorithm for task scheduling in cloud computing. In: 2018 2nd International Conference on Trends in Electronics and Informatics (ICOEI), IEEE, pp. 780–784 (2018)

  48. Dai, Y., Lou, Y., Lu, X.: A task scheduling algorithm based on genetic algorithm and ant colony optimization algorithm with multi-qos constraints in cloud computing. In: 2015 7th International Conference on Intelligent Human-Machine Systems and Cybernetics, vol. 2, IEEE, pp. 428–431 (2015)

  49. Abubakar, A., Yahaya, A.: Task scheduling in cloud computing environment using particle swarm optimization algorithm. Niger. J. Sci. Res. 14, 106 (2015)

    Google Scholar 

  50. Liu, Y., Shu, W., Zhang, C.: A parallel task scheduling optimization algorithm based on clonal operator in green cloud computing. J. Commun. 11, 185–191 (2016)

    Google Scholar 

  51. Loo, S.M., Wells, B.E.: Task scheduling in a finite-resource, reconfigurable hardware/software codesign environment. INFORMS J. Comput. 18, 151–172 (2006)

    MATH  Google Scholar 

  52. Saranu, K., Jaganathan, S.: Intensified scheduling algorithm for virtual machine tasks in cloud computing. In: Artificial Intelligence and Evolutionary Algorithms in Engineering Systems. Springer, Cham, pp. 283–290 (2015)

  53. Al-Rahayfeh, A., Atiewi, S., Abuhussein, A., Almiani, M.: Novel approach to task scheduling and load balancing using the dominant sequence clustering and mean shift clustering algorithms. Future Internet 11, 109 (2019)

    Google Scholar 

  54. Abdi, S., Motamedi, S.A., Sharifian, S.: Task scheduling using modified PSO algorithm in cloud computing environment. In: International Conference on Machine Learning, Electrical and Mechanical Engineering, pp. 8–9 (2014)

  55. Ks, S.R., Murugan, S.: Memory based hybrid dragonfly algorithm for numerical optimization problems. Expert Syst. Appl. 83, 63–78 (2017)

    Google Scholar 

  56. Suresh, V., Sreejith, S.: Generation dispatch of combined solar thermal systems using dragonfly algorithm. Computing 99, 59–80 (2017)

    MathSciNet  MATH  Google Scholar 

  57. Alshinwan, M., Abualigah, L., Shehab, M., Abd Elaziz, M., Khasawneh, A.M., Alabool, H., Al Hamad, H.: Dragonfly algorithm: a comprehensive survey of its results, variants, and applications. Multimedia Tools Appl. (2021). https://doi.org/10.1007/s11042-020-10255-3

    Article  Google Scholar 

  58. Alyasseri, Z.A.A., Khader, A.T., Al-Betar, M.A., Awadallah, M.A.: Hybridizing \(\beta\)-hill climbing with wavelet transform for denoising ESG signals. Inf. Sci. 429, 229–246 (2018)

    Google Scholar 

  59. Humane, P., Varshapriya, J.: Simulation of cloud infrastructure using cloudsim simulator: a practical approach for researchers. In: 2015 International Conference on Smart Technologies and Management for Computing, Communication, Controls, Energy and Materials (ICSTM), IEEE, pp. 207–211 (2015)

  60. Zhang, L., Li, K., Li, C., Li, K.: Bi-objective workflow scheduling of the energy consumption and reliability in heterogeneous computing systems. Inf. Sci. 379, 241–256 (2017)

    Google Scholar 

  61. Mirjalili, S., Lewis, A.: The whale optimization algorithm. Adv. Eng. Softw. 95, 51–67 (2016)

    Google Scholar 

  62. Zuo, X., Zhang, G., Tan, W.: Self-adaptive learning PSO-based deadline constrained task scheduling for hybrid IAAS cloud. IEEE Trans. Autom. Sci. Eng. 11, 564–573 (2013)

    Google Scholar 

  63. Mirjalili, S.: The ant lion optimizer. Adv. Eng. Softw. 83, 80–98 (2015)

    Google Scholar 

  64. Feitelson, D.G., Tsafrir, D., Krakov, D.: Experience with using the parallel workloads archive. J. Parallel Distrib. Comput. 74, 2967–2982 (2014)

    Google Scholar 

  65. Meng, J., McCauley, S., Kaplan, F., Leung, V.J., Coskun, A.K.: Simulation and optimization of HPC job allocation for jointly reducing communication and cooling costs. Sustain. Computi. Inf. Syst. 6, 48–57 (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Faculty of Computer Sciences and Informatics, Amman Arab University, Amman, 11953, Jordan

    Laith Abualigah

  2. School of Computer Sciences, Universiti Sains Malaysia (USM), Gelugor, 11800, Penang, Malaysia

    Laith Abualigah

  3. Division of Engineering, New York University Abu Dhabi, Saadiyat Island, 129188, Abu Dhabi, United Arab Emirates

    Ali Diabat

  4. Department of Civil and Urban Engineering, Tandon School of Engineering, New York University, Brooklyn, NY, 11201, USA

    Ali Diabat

  5. Department of Mathematics, Faculty of Science, Zagazig University, Zagazig, 44519, Egypt

    Mohamed Abd Elaziz

Authors
  1. Laith Abualigah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ali Diabat
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohamed Abd Elaziz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Laith Abualigah.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest regarding the publication of this paper.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Abualigah, L., Diabat, A. & Elaziz, M.A. Intelligent workflow scheduling for Big Data applications in IoT cloud computing environments. Cluster Comput 24, 2957–2976 (2021). https://doi.org/10.1007/s10586-021-03291-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-021-03291-7

Keywords

Search

Navigation