Skip to main content
SpringerLink
Log in

A Spark-based differential evolution with grouping topology model for large-scale global optimization

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

Over the past few years, cloud computing model (e.g., Spark) has aroused huge attention. Differential evolution (DE) has been applied to cloud computing models by a number of researchers for its merits in solving large-scale global optimization problems (LSGO), and remarkable results have been achieved. Moreover, we noticed that a combination of better topology and migration strategy is critical to solve the mentioned problems when DE algorithm acts as an internal optimizer for Spark cloud computing model. However, rare studies have been conducted to combine the combination to enhance the performance of DE algorithm for solving large-scale global optimization problems. Thus, inspired by the mentioned insights, we propose a novel grouping topology model that uses DE variants as internal optimizers to solve LSGO problems, called SgtDE. In SgtDE, population is split into subgroups evenly, and various topology structures are introduced to migrate individuals between and within subgroups. In this paper, five types of DE are adopted as the internal optimizers. By comparing the 20 benchmark functions presented on CEC2010, the results demonstrate that the SgtDE, especially a combination of better topology and migration strategy, exhibits significant performance in applying various DE variants. Thus, the SgtDE can act as the next generation optimizer of the cloud computing platform.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Ali, M., Pant, M.: Improving the performance of differential evolution algorithm using cauchy mutation. Soft Comput. 15(5), 991–1007 (2011)

    Article  Google Scholar 

  2. Ali, M.Z., Awad, N.H., Suganthan, P.N.: Multi-population differential evolution with balanced ensemble of mutation strategies for large-scale global optimization. Appl. Soft Comput. 33, 304–327 (2015)

    Article  Google Scholar 

  3. Arnaldo, I., Contreras, I., Millán-Ruiz, D., Hidalgo, J.I., Krasnogor, N.: Matching island topologies to problem structure in parallel evolutionary algorithms. Soft Comput. 17(7), 1209–1225 (2013)

    Article  Google Scholar 

  4. Balabanov, T., Zankinski, I., Barova, M.: Strategy for individuals distribution by incident nodes participation in star topology of distributed evolutionary algorithms. Cybern. Inf. Technol. 16(1), 80–88 (2016)

    Google Scholar 

  5. Brest, J., Greiner, S., Boskovic, B., Mernik, M., Zumer, V.: Self-adapting control parameters in differential evolution: a comparative study on numerical benchmark problems. IEEE Trans. Evol. Comput. 10(6), 646–657 (2006)

    Article  Google Scholar 

  6. Brest, J., Zamuda, A., Fister, I., Maučec, M.S.: Large scale global optimization using self-adaptive differential evolution algorithm. In: Proceedings of the IEEE Congress on Evolutionary Computation, pp. 1–8. IEEE (2010)

  7. Brest, J., Zamuda, A., Fister, I., Maučec, M.S., et al.: Self-adaptive differential evolution algorithm with a small and varying population size. In: Proceedings of the 2012 IEEE Congress on Evolutionary Computation, pp. 1–8. IEEE (2012)

  8. Chen, W., Weise, T., Yang, Z., Tang, K.: Large-scale global optimization using cooperative coevolution with variable interaction learning. In: Proceedings of the International Conference on Parallel Problem Solving from Nature, pp. 300–309. Springer (2010)

  9. Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)

    Article  Google Scholar 

  10. Dong, W., Wang, Y., Zhou, M.: A latent space-based estimation of distribution algorithm for large-scale global optimization. Soft Comput. 23(13), 4593–4615 (2019)

    Article  Google Scholar 

  11. Eberhart, R., Kennedy, J.: Particle swarm optimization. In: Proceedings of the IEEE international conference on neural networks, vol. 4, pp. 1942–1948. Citeseer (1995)

  12. Ge, Y., Yu, W., Lin, Y., Gong, Y., Zhan, Z., Chen, W., Zhang, J.: Distributed differential evolution based on adaptive mergence and split for large-scale optimization. IEEE Trans. Cybern. 48(7), 2166–2180 (2017)

    Article  Google Scholar 

  13. Guo, Z., Yang, H., Wang, S., Zhou, C., Liu, X.: Adaptive harmony search with best-based search strategy. Soft Comput. 22(4), 1335–1349 (2018)

    Article  Google Scholar 

  14. Harik, G.R., Lobo, F.G., Goldberg, D.E.: The compact genetic algorithm. IEEE Trans. Evol. Comput. 3(4), 287–297 (1999)

    Article  Google Scholar 

  15. Lopes, R.A., de Freitas, A.R.: Island-cellular model differential evolution for large-scale global optimization. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 1841–1848. ACM (2017)

  16. Lopes, R.A., Silva, R.C.P., Campelo, F., Guimaraes, F.G.: A multi-agent approach to the adaptation of migration topology in island model evolutionary algorithms. In: Proceedings of the 2012 Brazilian Symposium on Neural Networks, pp. 160–165. IEEE (2012)

  17. Lopes, R.A., Pedrosa Silva, R.C., Freitas, A.R., Campelo, F., Guimarães, F.G.: A study on the configuration of migratory flows in island model differential evolution. In: Proceedings of the Companion Publication of the 2014 Annual Conference on Genetic and Evolutionary Computation, pp. 1015–1022. ACM (2014)

  18. Mahdavi, S., Shiri, M.E., Rahnamayan, S.: Cooperative co-evolution with a new decomposition method for large-scale optimization. In: Proceedings of the 2014 IEEE Congress on Evolutionary Computation (CEC), pp. 1285–1292. IEEE (2014)

  19. Mallipeddi, R., Suganthan, P.N., Pan, Q.K., Tasgetiren, M.F.: Differential evolution algorithm with ensemble of parameters and mutation strategies. Appl. Soft Comput. 11(2), 1679–1696 (2011)

    Article  Google Scholar 

  20. Muelas, S., La Torre, A., Peña, J.M.: A memetic differential evolution algorithm for continuous optimization. In: Proceedings of the 2009 Ninth International Conference on Intelligent Systems Design and Applications, pp. 1080–1084. IEEE (2009)

  21. Omidvar, M.N., Li, X., Yang, Z., Yao, X.: Cooperative co-evolution for large scale optimization through more frequent random grouping. In: Proceedings of the IEEE Congress on Evolutionary Computation, pp. 1–8. IEEE (2010)

  22. Omidvar, M.N., Li, X., Yao, X.: Cooperative co-evolution with delta grouping for large scale non-separable function optimization. In: Proceedings of the IEEE Congress on Evolutionary Computation, pp. 1–8. IEEE (2010)

  23. Peng, H., Tan, X., Deng, C., Peng, S.: Sparkcude: a spark-based differential evolution for large-scale global optimisation. Int. J. High Perform. Syst. Arch. 7(4), 211–222 (2017)

    Google Scholar 

  24. Peng, H., Wu, Z., Deng, C.: Enhancing differential evolution with commensal learning and uniform local search. Chin. J. Electron. 26(4), 725–733 (2017)

    Article  Google Scholar 

  25. Peng, H., Guo, Z., Deng, C., Wu, Z.: Enhancing differential evolution with random neighbors based strategy. J. Comput. Sci. 26, 501–511 (2018)

    Article  MathSciNet  Google Scholar 

  26. Peng, H., Deng, C., Wu, Z.: Spbso: self-adaptive brain storm optimization algorithm with pbest guided step-size. J. Intell. Fuzzy Syst. 36(6), 5423–5434 (2019)

    Article  Google Scholar 

  27. Potter, M.A., De Jong, K.A.: A cooperative coevolutionary approach to function optimization. In: Proceedings of the International Conference on Parallel Problem Solving from Nature, pp. 249–257. Springer (1994)

  28. Qin, A.K., Suganthan, P.N.: Self-adaptive differential evolution algorithm for numerical optimization. In: Proceedings of the 2005 IEEE congress on evolutionary computation, vol. 2, pp. 1785–1791. IEEE (2005)

  29. Rahnamayan, S., Tizhoosh, H.R., Salama, M.M.: Opposition-based differential evolution. IEEE Trans. Evol. Comput. 12(1), 64–79 (2008)

    Article  Google Scholar 

  30. Segura, C., Coello, C.A.C., Hernández-Díaz, A.G.: Improving the vector generation strategy of differential evolution for large-scale optimization. Inf. Sci. 323, 106–129 (2015)

    Article  MathSciNet  Google Scholar 

  31. Skolicki, Z., De Jong, K.: The influence of migration sizes and intervals on island models. In: Proceedings of the 7th annual conference on Genetic and evolutionary computation, pp. 1295–1302. ACM (2005)

  32. Storn, R., Price, K.: Differential evolution—a simple and efficient heuristic for global optimization over continuous spaces. J. Glob. Optim. 11(4), 341–359 (1997)

    Article  MathSciNet  Google Scholar 

  33. Sun, J., Dong, H.: Cooperative co-evolution with correlation identification grouping for large scale function optimization. In: Proceedings of the 2013 IEEE Third International Conference on Information Science and Technology (ICIST), pp. 889–893. IEEE (2013)

  34. Tan, X.J., Deng, C.S., Dong, X.G., Yuan, S.H., Wu, Z., Peng, H.: Sparkde:a parallel version of differential evolution based on resilient distributed datasets model in cloud computing. Comput. Sci. 43(9), 116–119 (2016)

    Google Scholar 

  35. Tang, K., Li, X., Suganthan, P., Yang, Z., Weise, T.: Benchmark functions for the cec’2010 special session and competition on large-scale global optimization. Nature Inspired Computation and Applications Laboratory, USTC, Tech. rep., China, pp. 1–23 (2010)

  36. Teijeiro, D., Pardo, X.C., González, P., Banga, J.R., Doallo, R.: Implementing parallel differential evolution on spark. In: Proceedings of the European Conference on the Applications of Evolutionary Computation, pp. 75–90. Springer (2016)

  37. Van den Bergh, F., Engelbrecht, A.P.: A cooperative approach to particle swarm optimization. IEEE Trans. Evol. Comput. 8(3), 225–239 (2004)

    Article  Google Scholar 

  38. Veronese, L.d.P., Krohling, R.A.: Differential evolution algorithm on the gpu with c-cuda. In: Proceedings of the IEEE Congress on Evolutionary Computation, pp. 1–7. IEEE (2010)

  39. Wang, H., Wu, Z., Rahnamayan, S., Jiang, D.: Sequential de enhanced by neighborhood search for large scale global optimization. In: Proceedings of the IEEE Congress on Evolutionary Computation, pp. 1–7. IEEE (2010)

  40. Wang, Y., Cai, Z., Zhang, Q.: Differential evolution with composite trial vector generation strategies and control parameters. IEEE Trans. Evol. Comput. 15(1), 55–66 (2011)

    Article  Google Scholar 

  41. Wang, H., Wu, Z., Rahnamayan, S.: Enhanced opposition-based differential evolution for solving high-dimensional continuous optimization problems. Soft Comput. 15(11), 2127–2140 (2011)

    Article  Google Scholar 

  42. Wang, Y., Cai, Z., Zhang, Q.: Enhancing the search ability of differential evolution through orthogonal crossover. Inf. Sci. 185(1), 153–177 (2012)

    Article  MathSciNet  Google Scholar 

  43. Wang, H., Rahnamayan, S., Wu, Z.: Parallel differential evolution with self-adapting control parameters and generalized opposition-based learning for solving high-dimensional optimization problems. J. Parallel Distrib. Comput. 73(1), 62–73 (2013)

    Article  Google Scholar 

  44. Weber, M., Neri, F., Tirronen, V.: Distributed differential evolution with explorative–exploitative population families. Genet. Program. Evol. Mach. 10(4), 343 (2009)

    Article  Google Scholar 

  45. Weber, M., Neri, F., Tirronen, V.: Shuffle or update parallel differential evolution for large-scale optimization. Soft Comput. 15(11), 2089–2107 (2011)

    Article  Google Scholar 

  46. Yang, Z., Tang, K., Yao, X.: Large scale evolutionary optimization using cooperative coevolution. Inf. Sci. 178(15), 2985–2999 (2008)

    Article  MathSciNet  Google Scholar 

  47. Yang, Z., Tang, K., Yao, X.: Multilevel cooperative coevolution for large scale optimization. In: Proceedings of the 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence), pp. 1663–1670. IEEE (2008)

  48. Yang, Z., Tang, K., Yao, X.: Scalability of generalized adaptive differential evolution for large-scale continuous optimization. Soft Comput. 15(11), 2141–2155 (2011)

    Article  Google Scholar 

  49. Yue, C., Qu, B., Liang, J.: A multi-objective particle swarm optimizer using ring topology for solving multimodal multiobjective problems. IEEE Trans. Evol. Comput. 22(5), 805–817 (2017)

    Article  Google Scholar 

  50. Zaharia, M., Chowdhury, M., Das, T., Dave, A., Ma, J., McCauley, M., Franklin, M.J., Shenker, S., Stoica, I.: Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In: Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, pp. 1–14. USENIX Association (2012)

  51. Zamuda, A., Brest, J., Boskovic, B., Zumer, V.: Large scale global optimization using differential evolution with self-adaptation and cooperative co-evolution. In: Proceedings of the 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence), pp. 3718–3725. IEEE (2008)

  52. Zhang, J., Sanderson, A.C.: Jade: adaptive differential evolution with optional external archive. IEEE Trans. Evol. Comput. 13(5), 945–958 (2009)

    Article  Google Scholar 

  53. Zhao, S.Z., Suganthan, P.N., Das, S.: Self-adaptive differential evolution with multi-trajectory search for large-scale optimization. Soft Comput. 15(11), 2175–2185 (2011)

    Article  Google Scholar 

  54. Zhou, X., Wu, Z., Wang, H.: Elite opposition-based differential evolution for solving large-scale optimization problems and its implementation on gpu. In: Proceedings of the 2012 13th International Conference on Parallel and Distributed Computing, Applications and Technologies, pp. 727–732. IEEE (2012)

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China (No.61763019), the Science and Technology Plan Projects of Jiangxi Provincial Education Department (No.GJJ180891).

Author information

Authors and Affiliations

  1. School of Information Science and Technology, Jiujiang University, Jiujiang, 332005, China

    Zhihui He, Hu Peng, Jianqiang Chen & Changshou Deng

  2. School of Computer Science, Wuhan University, Wuhan, 430072, China

    Zhijian Wu

Authors
  1. Zhihui He
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hu Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jianqiang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Changshou Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhijian Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hu Peng.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

He, Z., Peng, H., Chen, J. et al. A Spark-based differential evolution with grouping topology model for large-scale global optimization. Cluster Comput 24, 515–535 (2021). https://doi.org/10.1007/s10586-020-03124-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-020-03124-z

Keywords

Search

Navigation