Skip to main content

Advertisement

Springer Nature Link
Log in

A genetic mixed-integer optimization of neural network hyper-parameters

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Neural networks (NN) have become immensely popular for their effectiveness and flexibility in learning complicated patterns. Despite this success, they are often considered difficult to design because of the wide variety of parameters they require. Determining the most optimal selection of parameters has become tedious and costly, and neural architecture search (NAS) methods have been employed to try and take the guesswork out of the process. A common NAS approach is the genetic algorithm (GA); however, its usage is often exclusively tied to either the learnable parameters, or the meta-parameters that augment the learning. This work proposes an experimental approach for optimizing both real-valued weights and discrete meta-parameters simultaneously. Experimental results have shown that the current approach evolves both parameter sets effectively for simple problems like Iris, but still struggles in finding an optimal model for more rigorous problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Abadi M et al (2015) TensorFlow: arge-scale machine learning on heterogeneous systems. https://www.tensorflow.org

  2. Abed-alguni BH (2019) Island-based cuckoo search with highly disruptive polynomial mutation. Int J Artif Intell 17(1):57–82

    Google Scholar 

  3. Berner C, Brockman G, Chan B et al (2019) Dota 2 with large scale deep reinforcement learning. arXiv:1912.06680

  4. Blank J, Deb K (2020) pymoo: multi-objective optimization in python. IEEE Access 8:89,497-89,509. https://doi.org/10.1109/ACCESS.2020.2990567

    Article  Google Scholar 

  5. Caltagirone L, Bellone M, Svensson L et al (2019) Lidar-camera fusion for road detection using fully convolutional neural networks. Robot Auton Syst 111:125–131. https://doi.org/10.1016/j.robot.2018.11.002

    Article  Google Scholar 

  6. Carvalho ED, Silva RR, Araújo FH et al (2021) An approach to the classification of covid-19 based on ct scans using convolutional features and genetic algorithms. Comput Biol Med 136(104):744. https://doi.org/10.1016/j.compbiomed.2021.104744

    Article  Google Scholar 

  7. Tm C, Rz Y (2009) Integrating messy genetic algorithms and simulation to optimize resource utilization. Comput Aided Civ Infrastruct Eng 24(6):401–415. https://doi.org/10.1111/j.1467-8667.2008.00588.x

    Article  Google Scholar 

  8. Chollet F et al (2015) Keras. https://keras.io

  9. Datta L (2020) A survey on activation functions and their relation with xavier and he normal initialization. arXiv:2004.06632

  10. Deb K (2000) An efficient constraint handling method for genetic algorithms. Comput Methods Appl Mech Eng 186(2–4):311–338. https://doi.org/10.1016/S0045-7825(99)00389-8

    Article  MATH  Google Scholar 

  11. Deb K, Agrawal RB (1995) Simulated binary crossover for continuous search space. Complex Syst 9

  12. Deb K, Pratap A, Agarwal S et al (2002) A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Trans Evol Comput 6(2):182–197. https://doi.org/10.1109/4235.996017

    Article  Google Scholar 

  13. Deb K, Sindhya K, Okabe T (2007) Self-adaptive simulated binary crossover for real-parameter optimization. In: Proceedings of the 9th annual conference on genetic and evolutionary computation, pp 1187–1194. https://doi.org/10.1145/1276958.1277190

  14. Ding S, Su C, Yu J (2011) An optimizing bp neural network algorithm based on genetic algorithm. Artif Intell Rev 36(2):153–162. https://doi.org/10.1007/s10462-011-9208-z

    Article  Google Scholar 

  15. Goldberg DE, Holland JH (1988) Genetic algorithms and machine learning. Mach Learn 3:95–99. https://doi.org/10.1023/A:1022602019183

    Article  Google Scholar 

  16. Goldberg DE, Deb K, Kargupta H, et al (1993) Rapid, accurate optimization of difficult problems using messy genetic algorithms. In: Proceedings of the Fifth International Conference on Genetic Algorithms,(Urbana, USA), Proceedings of the Fifth International Conference on Genetic Algorithms, pp 59–64

  17. Gupta JN, Sexton RS (1999) Comparing backpropagation with a genetic algorithm for neural network training. Omega 27(6):679–684. https://doi.org/10.1016/S0305-0483(99)00027-4

    Article  Google Scholar 

  18. Harris CR et al (2020) Array programming with NumPy. Nature 585(7825):357–362. https://doi.org/10.1038/s41586-020-2649-2

    Article  Google Scholar 

  19. Hunter JD (2007) Matplotlib: a 2d graphics environment. Comput Sci Eng 9(3):90–95. https://doi.org/10.1109/MCSE.2007.55

    Article  Google Scholar 

  20. Kantardzic M (2011) Data mining: concepts, models, methods, and algorithms. Wiley, Amsterdam

    Book  Google Scholar 

  21. Keith JA, Vassilev-Galindo V, Cheng B et al (2021) Combining machine learning and computational chemistry for predictive insights into chemical systems. Chem Rev 121(16):9816–9872. https://doi.org/10.1021/acs.chemrev.1c00107

    Article  Google Scholar 

  22. Lam SK, Pitrou A, Seibert S (2015) Numba: A llvm-based python jit compiler. In: Proceedings of the second workshop on the llvm compiler infrastructure in HPC, pp 1–6. https://doi.org/10.1145/2833157.2833162

  23. Liashchynskyi P, Liashchynskyi P (2019) Grid search, random search, genetic algorithm: a big comparison for nas. arXiv:1912.06059

  24. Lipowski A, Lipowska D (2012) Roulette-wheel selection via stochastic acceptance. Phys A 391(6):2193–2196. https://doi.org/10.1016/j.physa.2011.12.004

    Article  Google Scholar 

  25. Masi F, Stefanou I, Vannucci P et al (2021) Thermodynamics-based artificial neural networks for constitutive modeling. J Mech Phys Solids 147(104):277. https://doi.org/10.1016/j.jmps.2020.104277

    Article  MathSciNet  MATH  Google Scholar 

  26. Miller GF, Todd PM, Hegde SU (1989) Designing neural networks using genetic algorithms. In: ICGA, pp 379–384

  27. Nikbakht S, Anitescu C, Rabczuk T (2021) Optimizing the neural network hyperparameters utilizing genetic algorithm. J Zhejiang Univ Sci A 22(6):407–426. https://doi.org/10.1631/jzus.A2000384

    Article  Google Scholar 

  28. NVIDIA, Vingelmann P, Fitzek FH (2020) Cuda, release: 10.2.89. https://developer.nvidia.com/cuda-toolkit

  29. O’Shea K, Nash R (2015) An introduction to convolutional neural networks. arXiv:1511.08458

  30. Pedregosa F et al (2011) Scikit-learn: machine learning in python. J Mach Learn Res 12:2825–2830

    MathSciNet  MATH  Google Scholar 

  31. Pham TD (2021) Classification of covid-19 chest X-rays with deep learning: new models or fine tuning? Health Inf Sci Syst 9(1):1–11. https://doi.org/10.1007/s13755-020-00135-3

    Article  Google Scholar 

  32. Raschka S, Mirjalili V (2017) Python machine learning: machine learning and deep learning with python. Scikit-Learn, and TensorFlow Second edition ed

  33. Real E, Moore S, Selle A, et al (2017) Large-scale evolution of image classifiers. In: Proceedings of the 34th international conference on machine learning, proceedings of machine learning research, vol 70. PMLR, pp 2902–2911

  34. Shinners P (2011) Pygame. http://www.pygame.org

  35. Silver D, Hubert T, Schrittwieser J, et al (2017) Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv:1712.01815

  36. Smith LN (2018) A disciplined approach to neural network hyper-parameters: part 1–learning rate, batch size, momentum, and weight decay. arXiv:1803.09820

  37. Wirsansky E (2020) Hands-on genetic algorithms with python: applying genetic algorithms to solve real-world deep learning and artificial intelligence problems. Packt Publishing Ltd, Brimingham

    Google Scholar 

  38. Xiao X, Yan M, Basodi S, et al (2018) Efficient hyperparameter optimization in deep learning using a variable length genetic algorithm. arXiv:1803.09820

  39. Yu X, Gen M (2010) Introduction to evolutionary algorithms. Springer, New York

    Book  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Engineering and Computer Science, Morehead State University, 150 University Blvd, Morehead, KY, 40351, USA

    Kyle Spurlock & Heba Elgazzar

Authors
  1. Kyle Spurlock
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Heba Elgazzar
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Heba Elgazzar.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Code, Data Availability

All code, data, and results described in this work are available at https://github.com/kspurlock/GAAP.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Spurlock, K., Elgazzar, H. A genetic mixed-integer optimization of neural network hyper-parameters. J Supercomput 78, 14680–14702 (2022). https://doi.org/10.1007/s11227-022-04475-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-022-04475-7

Keywords