Skip to main content
SpringerLink
Log in

A Novel Two-Level Clustering-Based Differential Evolution Algorithm for Training Neural Networks

  • Conference paper
  • First Online:
Applications of Evolutionary Computation (EvoApplications 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14634))

  • 161 Accesses

Abstract

Determining appropriate weights and biases for feed-forward neural networks is a critical task. Despite the prevalence of gradient-based methods for training, these approaches suffer from sensitivity to initial values and susceptibility to local optima. To address these challenges, we introduce a novel two-level clustering-based differential evolution approach, C2L-DE, to identify the initial seed for a gradient-based algorithm. In the initial phase, clustering is employed to detect some regions in the search space. Population updates are then executed based on the information available within each region. A new central point is proposed in the subsequent phase, leveraging cluster centres for incorporation into the population. Our C2L-DE algorithm is compared against several recent DE-based neural network training algorithms, and is shown to yield favourable performance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Notes

  1. 1.

    https://archive.ics.uci.edu/ml/index.php.

References

  1. Abrishami, N., Sepaskhah, A.R., Shahrokhnia, M.H.: Estimating wheat and maize daily evapotranspiration using artificial neural network. Theor. Appl. Climatol. 135(3), 945–958 (2018). https://doi.org/10.1007/s00704-018-2418-4

    Article  Google Scholar 

  2. Amirsadri, S., Mousavirad, S.J., Ebrahimpour-Komleh, H.: A levy flight-based grey wolf optimizer combined with back-propagation algorithm for neural network training. Neural Comput. Appl. 30(12), 3707–3720 (2017). https://doi.org/10.1007/s00521-017-2952-5

    Article  Google Scholar 

  3. Bojnordi, E., Mousavirad, S.J., Pedram, M., et al.: Improving the generalisation ability of neural networks using a Lévy flight distribution algorithm for classification problems. New Gener. Comput. 41(2), 225–242 (2023). https://doi.org/10.1007/s00354-023-00214-5

    Article  Google Scholar 

  4. Bojnordi,E., Mousavirad,S.J., Schaefer, G., Korovin, I.: MCS-HMS: a multi-cluster selection strategy for the human mental search algorithm. In: IEEE Symposium Series on Computational Intelligence, pp. 1–6, 2021

    Google Scholar 

  5. Cai, Z., Gong, W., Ling, C.X., Zhang, H.: A clustering-based differential evolution for global optimization. Appl. Soft Comput. 11(1), 1363–1379 (2011)

    Article  Google Scholar 

  6. Deb, K.: A population-based algorithm-generator for real-parameter optimization. Soft. Comput. 9(4), 236–253 (2005)

    Article  Google Scholar 

  7. Ding, S., Chunyang, S., Junzhao, Yu.: An optimizing BP neural network algorithm based on genetic algorithm. Artif. Intell. Rev. 36(2), 153–162 (2011)

    Article  Google Scholar 

  8. Duan, H., Huang, L.: Imperialist competitive algorithm optimized artificial neural networks for UCAV global path planning. Neurocomputing 125, 166–171 (2014)

    Article  Google Scholar 

  9. El-Bakry, M.Y., El-Dahshan, E.-S.A., El-Hamied, E.F.A.: Charged particle pseudorapidity distributions for Pb-Pb and Au-Au collisions using neural network model. Ukrainian J. Phys. 58(8), 709–709 (2013)

    Article  Google Scholar 

  10. Fister, I., Fister, D., Deb, S., Mlakar, U., Brest, J.: Post hoc analysis of sport performance with differential evolution. Neural Comput. Appl. 32, 1–10 (2018)

    Google Scholar 

  11. Hosaka, T.: Bankruptcy prediction using imaged financial ratios and convolutional neural networks. Expert Syst. Appl. 117, 287–299 (2019)

    Article  Google Scholar 

  12. Ilonen, J., Kamarainen, J.-K., Lampinen, J.: Differential evolution training algorithm for feed-forward neural networks. Neural Process. Lett. 17(1), 93–105 (2003)

    Article  Google Scholar 

  13. Kim, D., Kim, H., Chung, D.: A modified genetic algorithm for fast training neural networks. In: International Symposium on Neural Networks, pp. 660–665 (2005)

    Google Scholar 

  14. Lera, G., Pinzolas, M.: Neighborhood based Levenberg-Marquardt algorithm for neural network training. IEEE Trans. Neural Networks 13(5), 1200–1203 (2002)

    Article  Google Scholar 

  15. Levenberg, K.: A method for the solution of certain non-linear problems in least squares. Q. Appl. Math. 2(2), 164–168 (1944)

    Article  MathSciNet  Google Scholar 

  16. MacQueen, J.: Some methods for classification and analysis of multivariate observations. In: 5th Berkeley Symposium on Mathematical Statistics and Probability, pp. 281–297 (1967)

    Google Scholar 

  17. Mandal, S., Saha, G., Pal, R.K.: Neural network training using firefly algorithm. Glob. J. Adv. Eng. Sci. 1(1), 7–11 (2015)

    Google Scholar 

  18. Marquardt, D.W.: An algorithm for least-squares estimation of nonlinear parameters. J. Soc. Ind. Appl. Math. 11(2), 431–441 (1963)

    Article  MathSciNet  Google Scholar 

  19. Mirjalili, S.: How effective is the Grey Wolf optimizer in training multi-layer perceptrons. Appl. Intell. 43(1), 150–161 (2015). https://doi.org/10.1007/s10489-014-0645-7

    Article  Google Scholar 

  20. Moravvej, S.V., Mousavirad, S.J., Oliva, D., Schaefer, G., Sobhaninia, Z.: An improved DE algorithm to optimise the learning process of a BERT-based plagiarism detection model. In: IEEE Congress on Evolutionary Computation, pp. 1–7 (2022)

    Google Scholar 

  21. Mousavirad, S.J., Bidgoli, A.A., Rahnamayan, S.: Tackling deceptive optimization problems using opposition-based DE with center-based Latin hypercube initialization. In: 14th International Conference on Computer Science and Education (2019)

    Google Scholar 

  22. Mousavirad, S.J., Bidgoli, A.A., Ebrahimpour-Komleh, H., G.S.: A memetic imperialist competitive algorithm with chaotic maps for multi-layer neural network training. Int. J. Bio-Inspired Comput. 14(4), 227–236 (2019)

    Google Scholar 

  23. Mousavirad, S.J., Bidgoli, A.A., Ebrahimpour-Komleh, H., Schaefer, G., Korovin, I.: An effective hybrid approach for optimising the learning process of multi-layer neural networks. In: International Symposium on Neural Networks, pp. 309–317 (2019)

    Google Scholar 

  24. Mousavirad, S.J., Gandomi, A.H., Homayoun, H.: A clustering-based differential evolution boosted by a regularisation-based objective function and a local refinement for neural network training. In: IEEE Congress on Evolutionary Computation, pp. 1–8 (2022)

    Google Scholar 

  25. Mousavirad, S.J., Jalali, S.M.J., Sajad, A., Abbas, K., Schaefer, G., Nahavandi, S.: Neural network training using a biogeography-based learning strategy. In: International Conference on Neural Information Processing (2020)

    Google Scholar 

  26. Mousavirad, S.J., Oliva, D., Hinojosa, S., Schaefer, G.: Differential evolution-based neural network training incorporating a centroid-based strategy and dynamic opposition-based learning. In: IEEE Congress on Evolutionary Computation, pp. 1233–1240 (2021)

    Google Scholar 

  27. Mousavirad, S.J., Rahmani, R., Dolatabadi, N.: A transfer learning based artificial neural network in geometrical design of textured surfaces for tribological applications. Surf. Topogr. Metrol. Prop. 11(2), 025001 (2023)

    Article  Google Scholar 

  28. Mousavirad, S.J., Rahnamayan, S.: Evolving feedforward neural networks using a quasi-opposition-based differential evolution for data classification. In: IEEE Symposium Series on Computational Intelligence (2020)

    Google Scholar 

  29. Mousavirad, S.J., Rahnamayan, S.: A novel center-based differential evolution algorithm. In: Congress on Evolutionary Computation (2020)

    Google Scholar 

  30. Mousavirad, S.J., Schaefer, G., Jalali, S.M.J., Korovin, I.: A benchmark of recent population-based metaheuristic algorithms for multi-layer neural network training. In: Genetic and Evolutionary Computation Conference Companion, pp. 1402–1408 (2020)

    Google Scholar 

  31. Mousavirad, S.J., Schaefer, G., Korovin, I.: An effective approach for neural network training based on comprehensive learning. In: International Conference on Pattern Recognition (2020)

    Google Scholar 

  32. Mousavirad, S.J., Schaefer, G., Korovin, I., Oliva, D.: RDE-OP: a region-based differential evolution algorithm incorporation opposition-based learning for optimising the learning process of multi-layer neural networks. In: Castillo, P.A., Jiménez Laredo, J.L. (eds.) EvoApplications 2021. LNCS, vol. 12694, pp. 407–420. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-72699-7_26

    Chapter  Google Scholar 

  33. Munkhdalai, L., Lee, J.Y., Ryu, K.H.: A hybrid credit scoring model using neural networks and logistic regression. In: Pan, J.-S., Li, J., Tsai, P.-W., Jain, L.C. (eds.) Advances in Intelligent Information Hiding and Multimedia Signal Processing. SIST, vol. 156, pp. 251–258. Springer, Singapore (2020). https://doi.org/10.1007/978-981-13-9714-1_27

    Chapter  Google Scholar 

  34. Nawi, N.M., khan, A., Rehman, M.Z., Aziz, M.A., Herawan, T., Abawajy, J.H.: An accelerated particle swarm optimization based Levenberg Marquardt back propagation algorithm. In: Loo, C.K., Yap, K.S., Wong, K.W., Teoh, A., Huang, K. (eds.) ICONIP 2014. LNCS, vol. 8835, pp. 245–253. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-12640-1_30

    Chapter  Google Scholar 

  35. Pedram, M., Mousavirad, S.J., Schaefer, G.: Training neural networks with Lévy flight distribution algorithm. In: 7th International Conference on Harmony Search, Soft Computing and Applications, pp. 93–103 (2022)

    Google Scholar 

  36. Rahmani, S., Mousavirad, S.J., El-Abd, M., Schaefer, G., Oliva, D.: Centroid-based differential evolution with composite trial vector generation strategies for neural network training. In: Correia, J., Smith, S., Qaddoura, R. (eds.) International Conference on the Applications of Evolutionary Computation, vol. 13989, pp. 608–622. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-30229-9_39

  37. Sexton, R.S., Gupta, J.N.D.: Comparative evaluation of genetic algorithm and backpropagation for training neural networks. Inform. Sci. 129(1–4), 45–59 (2000)

    Article  Google Scholar 

  38. Shi, Y., Eberhart, R.: A modified particle swarm optimizer. In: IEEE International Conference on Evolutionary Computation, pp. 69–73 (1998)

    Google Scholar 

  39. Si, T., Dutta, R.: Partial opposition-based particle swarm optimizer in artificial neural network training for medical data classification. Int. J. Inform. Technol. Decis. Making 18(5), 1717–1750 (2019)

    Article  Google Scholar 

  40. Storn, R., Price, K.: Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces. J. Global Optim. 11(4), 341–359 (1997)

    Article  MathSciNet  Google Scholar 

  41. Wang, X., et al.: Massive expansion and differential evolution of small heat shock proteins with wheat (triticum aestivum l.) polyploidization. Sci. Rep. 7(1), 1–12 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Mid Sweden University, Sundsvall, Sweden

    Seyed Jalaleddin Mousavirad

  2. Depto. de Ingeniería Electro -Fotónica, Universidad de Guadalajara, CUCEI, Guadalajara, Mexico

    Diego Oliva

  3. Department of Computer Science, Loughborough University, Loughborough, UK

    Gerald Schaefer

  4. Mälardalens University, Västerås, Sweden

    Mahshid Helali Moghadam

  5. College of Engineering and Applied Sciences, American University of Kuwait, Salmiya, Kuwait

    Mohammed El-Abd

Authors
  1. Seyed Jalaleddin Mousavirad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Diego Oliva
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gerald Schaefer
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mahshid Helali Moghadam
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Mohammed El-Abd
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Seyed Jalaleddin Mousavirad .

Editor information

Editors and Affiliations

  1. University of York, York, UK

    Stephen Smith

  2. University of Coimbra, Coimbra, Portugal

    João Correia

  3. University of Málaga, Málaga, Spain

    Christian Cintrano

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Mousavirad, S.J., Oliva, D., Schaefer, G., Moghadam, M.H., El-Abd, M. (2024). A Novel Two-Level Clustering-Based Differential Evolution Algorithm for Training Neural Networks. In: Smith, S., Correia, J., Cintrano, C. (eds) Applications of Evolutionary Computation. EvoApplications 2024. Lecture Notes in Computer Science, vol 14634. Springer, Cham. https://doi.org/10.1007/978-3-031-56852-7_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-56852-7_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-56851-0

  • Online ISBN: 978-3-031-56852-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation