Skip to main content
SpringerLink
Log in

Spark-based cooperative coevolution for large scale global optimization

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

The cooperative coevolution framework was introduced to address the shortcomings of metaheuristic algorithms in solving continuous large-scale global optimization problems. By dividing the problem into subcomponents which can be optimized separately, the framework can improve on both the solution’s quality as well as the computational speed by exposing a degree of parallelism. Distributed computing platforms, such as Apache Spark, have long been used to improve the speed of different algorithms in solving computational problems. This work proposes a distributed implementation of the cooperative coevolution framework for solving large-scale global optimization problems on the Apache Spark distributed computing platform. By using a formerly outlined distributed variant of the cooperative coevolution framework, features of the Spark platform are utilized to enhance the computational speed of the algorithm while maintaining comparable search quality to other works in the literature. To test for the proposed implementation’s improvement in computational speed, the CEC 2010 large-scale global optimization benchmark functions are used due to the diversity they offer in terms of complexity, separability and modality. Results of the proposed distributed implementation suggest that a speedup of up to ×3.36 is possible on large-scale global optimization benchmarks using the Apache Spark platform.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data availability

All data generated or analysed during this study are included in this published article.

References

  1. Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of ICNN’95-International Conference on Neural Networks, vol. 4, pp. 1942–1948 (1995). IEEE

  2. Goldberg, D.E., Holland, J.H.: Genetic algorithms and machine learning, vol. 3, pp. 95–99. Springer, London (1988)

    Google Scholar 

  3. Boussaïd, I., Lepagnot, J., Siarry, P.: A survey on optimization metaheuristics. Information Sci. 237, 82–117 (2013)

    Article  MathSciNet  Google Scholar 

  4. Hussain, K., Mohd Salleh, M.N., Cheng, S., Shi, Y.: Metaheuristic research: a comprehensive survey, vol. 52, pp. 2191–2233. Springer, London (2019)

    Google Scholar 

  5. Bellman, R.: Dynamic programming and lagrange multipliers. Proc. National Acad. Sci. U. S. A. 42(10), 767 (1956)

    Article  MathSciNet  Google Scholar 

  6. Omidvar, M.N., Li, X., Yao, X.: A review of population-based metaheuristics for large-scale black-box global optimization-Part I. IEEE Trans. Evolut. Comput. 26(5), 802–822 (2021)

    Article  Google Scholar 

  7. Omidvar, M.N., Li, X., Yao, X.: A review of population-based metaheuristics for large-scale black-box global optimization-Part II. IEEE Trans. Evolut. Comput. 26(5), 823–843 (2021)

    Article  Google Scholar 

  8. Chen, W.-N., Jia, Y.-H., Zhao, F., Luo, X.-N., Jia, X.-D., Zhang, J.: A cooperative co-evolutionary approach to large-scale multisource water distribution network optimization. IEEE Trans. Evolut. Comput. 23(5), 842–857 (2019)

    Article  Google Scholar 

  9. Sato, M., Fukuyama, Y., El-Abd, M., Iizaka, T., Matsui, T.: Total optimization of energy networks in smart city by cooperative coevolution using global-best brain storm optimization. In: 2019 IEEE Congress on Evolutionary Computation (CEC), pp. 681–688 (2019). IEEE

  10. Tan, B., Ma, H., Mei, Y., Zhang, M.: A cooperative coevolution genetic programming hyper-heuristics approach for on-line resource allocation in container-based clouds. IEEE Trans. Cloud Comput. 10(3), 1500–1514 (2020)

    Article  Google Scholar 

  11. Yang, Z., Tang, K., Yao, X.: Differential evolution for high-dimensional function optimization. In: 2007 IEEE Congress on Evolutionary Computation, pp. 3523–3530 (2007). IEEE

  12. Omidvar, M.N., Yang, M., Mei, Y., Li, X., Yao, X.: DG2: a faster and more accurate differential grouping for large-scale black-box optimization. IEEE Transa. Evolut. Comput. 21(6), 929–942 (2017)

    Article  Google Scholar 

  13. Gropp, W., Gropp, W.D., Lusk, E., Skjellum, A., Lusk, E.: Using MPI: portable parallel programming with the message-passing interface, vol. 1. MIT press, Cambridge (1999)

    Book  Google Scholar 

  14. Kelkawi, A., El-Abd, M., Ahmad, I.: GPU-based cooperative coevolution for large-scale global optimization. Neural Comput. Appl. 35(6), 4621–4642 (2023)

    Article  Google Scholar 

  15. Brodtkorb, A.R., Hagen, T.R., Sætra, M.L.: Graphics processing unit (GPU) programming strategies and trends in GPU computing. J. Parallel Distrib. Comput. 73(1), 4–13 (2013)

    Article  Google Scholar 

  16. Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I., et al.: Spark: cluster computing with working sets. HotCloud 10(10–10), 95 (2010)

    Google Scholar 

  17. Wang, S., Gao, B., Wang, K., Lauw, H.: Ccrank: Parallel learning to rank with cooperative coevolution. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 25, pp. 1249–1254 (2011)

  18. Cao, B., Zhao, J., Lv, Z., Liu, X.: A distributed parallel cooperative coevolutionary multiobjective evolutionary algorithm for large-scale optimization. IEEE Trans. Ind. Informatics 13(4), 2030–2038 (2017)

    Article  Google Scholar 

  19. Jia, Y.-H., Chen, W.-N., Gu, T., Zhang, H., Yuan, H.-Q., Kwong, S., Zhang, J.: Distributed cooperative co-evolution with adaptive computing resource allocation for large scale optimization. IEEE Trans. Evolut. Comput. 23(2), 188–202 (2018)

    Article  Google Scholar 

  20. He, Z., Peng, H., Chen, J., Deng, C., Wu, Z.: A spark-based differential evolution with grouping topology model for large-scale global optimization. Clust. Comput. 24, 515–535 (2021)

    Article  Google Scholar 

  21. Cao, B., Li, W., Zhao, J., Yang, S., Kang, X., Ling, Y., Lv, Z.: Spark-based parallel cooperative co-evolution particle swarm optimization algorithm. In: 2016 IEEE International Conference on Web Services (ICWS), pp. 570–577 (2016). IEEE

  22. Omidvar, M.N., Li, X., Yao, X.: Cooperative co-evolution with delta grouping for large scale non-separable function optimization. In: IEEE Congress on Evolutionary Computation, pp. 1–8 (2010). IEEE

  23. Chen, W., Weise, T., Yang, Z., Tang, K.: Large-scale global optimization using cooperative coevolution with variable interaction learning. In: International Conference on Parallel Problem Solving from Nature, pp. 300–309 (2010). Springer

  24. Omidvar, M.N., Li, X., Mei, Y., Yao, X.: Cooperative co-evolution with differential grouping for large scale optimization. IEEE Trans. Evolut. Comput. 18(3), 378–393 (2013)

    Article  Google Scholar 

  25. Yang, Z., Tang, K., Yao, X.: Self-adaptive differential evolution with neighborhood search. In: 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence), pp. 1110–1116 (2008). IEEE

  26. AlJame, M., Ahmad, I., Alfailakawi, M.: Apache spark implementation of whale optimization algorithm. Clust. Comput. 23(3), 2021–2034 (2020)

    Article  Google Scholar 

  27. Ma, X., Li, X., Zhang, Q., Tang, K., Liang, Z., Xie, W., Zhu, Z.: A survey on cooperative co-evolutionary algorithms. IEEE Trans. Evolut. Comput. 23(3), 421–441 (2018)

    Article  Google Scholar 

  28. Deb, K., Thiele, L., Laumanns, M., Zitzler, E.: Scalable test problems for evolutionary multiobjective optimization. Evolutionary multiobjective optimization, pp. 105–145. Springer, London (2005)

    Book  Google Scholar 

  29. Huband, S., Hingston, P., Barone, L., While, L.: A review of multiobjective test problems and a scalable test problem toolkit. IEEE Trans. Evolut. Compu. 10(5), 477–506 (2006)

    Article  Google Scholar 

  30. Firouznia, M., Ruiu, P., Trunfio, G.A.: Adaptive cooperative coevolutionary differential evolution for parallel feature selection in high-dimensional datasets. J. Supercomput. 10, 1–30 (2023)

    Google Scholar 

  31. Chen, Q., Sun, J., Palade, V.: Distributed contribution-based quantum-behaved particle swarm optimization with controlled diversity for large-scale global optimization problems. IEEE Access 7, 150093–150104 (2019)

    Article  Google Scholar 

  32. Teijeiro, D., Pardo, X.C., González, P., Banga, J.R., Doallo, R.: Implementing parallel differential evolution on spark. In: European Conference on the Applications of Evolutionary Computation, pp. 75–90 (2016). Springer

  33. Gong, Y.-J., Chen, W.-N., Zhan, Z.-H., Zhang, J., Li, Y., Zhang, Q., Li, J.-J.: Distributed evolutionary algorithms and their models: a survey of the state-of-the-art. Appl. Soft Comput. 34, 286–300 (2015)

    Article  Google Scholar 

  34. Tang, K., Li, X., Suganthan, P.N., Yang, Z., Weise, T.: Benchmark functions for the CEC’2010 special session and competition on large-scale global optimization. Technical report, Nature Inspired Computation and Applications Laboratory (2009)

  35. Dokeroglu, T., Sevinc, E., Kucukyilmaz, T., Cosar, A.: A survey on new generation metaheuristic algorithms. Comput. Indust. Eng. 137, 106040 (2019)

    Article  Google Scholar 

Download references

Funding

None.

Author information

Authors and Affiliations

  1. Department of Computer Engineering, College of Engineering and Petroleum, Kuwait University, Shedadiyah, Kuwait

    Ali Kelkawi & Imtiaz Ahmad

  2. College of Engineering and Applied Sciences, American University of Kuwait, Salmiya, Kuwait

    Mohammed El-Abd

Authors
  1. Ali Kelkawi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Imtiaz Ahmad
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohammed El-Abd
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Implementation, data collection and analysis were performed by AK. The first draft of the manuscript was written by AK and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Ali Kelkawi.

Ethics declarations

Conflict of interest

All authors declare that they have no conflicts of interest.

Informed consent

Not applicable.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kelkawi, A., Ahmad, I. & El-Abd, M. Spark-based cooperative coevolution for large scale global optimization. Cluster Comput 27, 1911–1926 (2024). https://doi.org/10.1007/s10586-023-04058-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-023-04058-y

Keywords

Search

Navigation