Skip to main content
Springer Nature Link
Log in

Research on Hadoop Task Scheduling Problem Based on Hybrid Whale Optimization-Genetic Algorithm

  • Conference paper
  • First Online:
Big Data (BigData 2021)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1496))

Included in the following conference series:

  • 1126 Accesses

Abstract

In the complex grid environment, how Hadoop’s scheduling tasks effectively use the shared available resources to complete the assigned tasks in the shortest time, which is a NP hard problem. In this paper, a Hybrid Whale Optimization-Genetic Algorithm (HWO-GA) algorithm is proposed to solve the task scheduling problem. The new HWO-GA algorithm introduces the mutation and crossover operator of Genetic Algorithm (GA) to overcome the defect that the traditional Whale Optimization Algorithm (WOA) is easy to fall into local optimal solution, so as to increase the global optimization ability of the algorithm. Experiments show that the HWO-GA algorithm has better convergence and optimization ability than the traditional WOA and GA algorithms, and can make more full use of shared resources.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Qin, X.P., Wang, H.-J.: Big Data analysis-competition and symbiosis of RDBMS and MapReduce. J. Softw. 23(1), 32–45 (2012)

    Article  Google Scholar 

  2. Lin, J.-C., Leu, F.-Y., Chen, Y.-P.: Impact of MapReduce policies on job completion reliability and job energy consumption. IEEE Trans. Parallel Distrib. Syst. 26(5), 1364–1378 (2015)

    Article  Google Scholar 

  3. Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)

    Article  Google Scholar 

  4. Wang, G., et al.: Behavioral simulations in MapReduce. Proc. VLDB Endowm. 3(1–2), 952–963 (2010)

    Article  Google Scholar 

  5. Mirjalili, S., Lewis, A.: The whale optimization algorithm. Adv. Eng. Softw. 95, 51–67 (2016)

    Article  Google Scholar 

  6. El Aziz, M.A., Ewees, A.A., Hassanien, A.E.: Whale optimization algorithm and moth-flame optimization for multilevel thresholding image segmentation. Exp. Syst. Appl. 83, 242–256 (2017)

    Article  Google Scholar 

  7. Muñoz, A., Rubio, F.: Evaluating genetic algorithms through the approximability hierarchy. J. Comput. Sci. 53, 101388 (2021)

    Article  MathSciNet  Google Scholar 

  8. Ongcunaruk, W., Ongkunaruk, P., Janssens, G.K.: Genetic algorithm for a delivery problem with mixed time windows. Comput. Industr. Eng. 159, 107478 (2021)

    Article  Google Scholar 

Download references

Acknowledgments

This work has been supported by projects of colleges and universities in Guang dong Province (No.2021ZDZX3016). Scientific research platforms and Young innovative talents project of colleges and universities in Guangdong Scientific research platforms and projects of colleges and universities in Guang Province (N2021KQNCX061)

Author information

Authors and Affiliations

  1. College of Computer, Guangdong University of Education, Guangzhou, 510303, Guangdong, China

    Jun Xu & JunFeng Peng

Authors
  1. Jun Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. JunFeng Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to JunFeng Peng .

Editor information

Editors and Affiliations

  1. National University of Defense Technology, Changsha, China

    Xiangke Liao

  2. Shenzhen University of Technology, Chinese Academy of Sciences, Shenzhen, China

    Wei Zhao

  3. University of Science and Technology of China, Hefei, China

    Enhong Chen

  4. Sun Yat-sen University, Guangzhou, China

    Nong Xiao

  5. Taiyuan University of Technology, Taiyuan, China

    Li Wang

  6. Nanjing University, Nanjing, China

    Yang Gao

  7. Nanjing University, Nanjing, China

    Yinghuan Shi

  8. Sun Yat-sen University, Guangzhou, China

    Changdong Wang

  9. Sun Yat-sen University, Guangzhou, China

    Dan Huang

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Xu, J., Peng, J. (2022). Research on Hadoop Task Scheduling Problem Based on Hybrid Whale Optimization-Genetic Algorithm. In: Liao, X., et al. Big Data. BigData 2021. Communications in Computer and Information Science, vol 1496. Springer, Singapore. https://doi.org/10.1007/978-981-16-9709-8_2

Download citation

  • DOI: https://doi.org/10.1007/978-981-16-9709-8_2

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-16-9708-1

  • Online ISBN: 978-981-16-9709-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships