Abstract
Convolutional neural networks (CNNs) are a very prevalent and powerful deep learning paradigm. In recent years, many neural architecture search (NAS) methods have been developed to automate the design process of CNN architectures, significantly reducing human effort. Among various search techniques, differential evolution (DE), as a popular evolutionary computation algorithm, has advantages of fewer control variables, fast convergence and powerful optimization capability. However, existing DE-based NAS methods simply use conventional search operators, and do not consider the global and local information in the search process well, thus failing to achieve satisfactory results. In this paper, we propose an eclectic DE approach for NAS that can make good use of the search capability of DE. The architectural parameters are encoded into two parts according to their ranges. A discrete mutation operator is proposed to evolve the part that has a small search space, while a versatile mutation operator is devised for the other part with a large search space. The proposed DE algorithm can well balance the global and local search, and yields better overall results than most compared methods with a single-path CNN architecture design based on basic operations on four benchmark image classification datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Awad, N., Mallik, N., Hutter, F.: Differential evolution for neural architecture search. arXiv preprint arXiv:2012.06400 (2020)
Bäck, T., Fogel, D.B., Michalewicz, Z.: Handbook of evolutionary computation. Release 97(1), B1 (1997)
Baymurzina, D., Golikov, E., Burtsev, M.: A review of neural architecture search. Neurocomputing 474, 82–93 (2022). https://doi.org/10.1016/j.neucom.2021.12.014
Chan, T.H., Jia, K., Gao, S., Lu, J., Zeng, Z., Ma, Y.: PCANet: a simple deep learning baseline for image classification? IEEE Trans. Image Process. 24(12), 5017–5032 (2015)
Ding, X., Zhang, X., Ma, N., Han, J., Ding, G., Sun, J.: RepVGG: making VGG-style convnets great again. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13733–13742 (2021)
Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: a survey. J. Mach. Learn. Res. 20(1), 1997–2017 (2019)
Gämperle, R., Müller, S.D., Koumoutsakos, P.: A parameter study for differential evolution. Adv. Intell. Syst. Fuzzy Syst. Evolut. Comput. 10(10), 293–298 (2002)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016)
Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)
Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, USA, pp. 2261–2269 (2017)
Huang, J., Xue, B., Sun, Y., Zhang, M.: A flexible variable-length particle swarm optimization approach to convolutional neural network architecture design. In: IEEE Congress on Evolutionary Computation (CEC), pp. 934–941 (2021). https://doi.org/10.1109/CEC45853.2021.9504716
Junior, F.E.F., Yen, G.G.: Particle swarm optimization of deep neural networks architectures for image classification. Swarm Evolut. Comput. 49, 62–74 (2019)
Kennedy, J., Eberhart, R.: Particle swarm optimization. In: International Conference on Neural Networks, vol. 4, pp. 1942–1948 (1995). https://doi.org/10.1109/ICNN.1995.488968
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. University of Toronto (2009)
Larochelle, H., Erhan, D., Courville, A., Bergstra, J., Bengio, Y.: An empirical evaluation of deep architectures on problems with many factors of variation. In: Proceedings of the 24th International Conference on Machine Learning, pp. 473–480 (2007)
Liu, Y., et al.: A survey on evolutionary neural architecture search. IEEE Trans. Neural Netw. Learn. Syst. (Early Access), 1–21 (2021). https://doi.org/10.1109/TNNLS.2021.3100554
Mi, J.X., Feng, J., Huang, K.Y.: Designing efficient convolutional neural network structure: a survey. Neurocomputing 489, 139–156 (2022)
Mitchell, M.: An Introduction to Genetic Algorithms. MIT Press, Cambridge (1998)
Real, E., et al.: Large-scale evolution of image classifiers. In: International Conference on Machine Learning (ICML), pp. 2902–2911 (2017)
Ruder, S.: An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747 (2016)
Salah, R., Vincent, P., Muller, X., et al.: Contractive auto-encoders: explicit invariance during feature extraction. In: International Conference on Machine Learning (ICML), pp. 833–840 (2011)
Shang, R., Zhu, S., Ren, J., Liu, H., Jiao, L.: Evolutionary neural architecture search based on evaluation correction and functional units. Knowl. Based Syst., 109206 (2022)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: International Conference on Learning Representations (ICLR) (2015)
Storn, R., Price, K.: Differential evolution - a simple and efficient heuristic for global optimization over continuous spaces. J. Glob. Optim. 11(4), 341–359 (1997)
Sun, Y., Xue, B., Zhang, M., Yen, G.G.: Evolving deep convolutional neural networks for image classification. IEEE Trans. Evolut. Comput. 24(2), 394–407 (2019)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (2018)
Szegedy, C., et al.: Going deeper with convolutions. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9 (2015)
Wang, B., Sun, Y., Xue, B., Zhang, M.: Evolving deep convolutional neural networks by variable-length particle swarm optimization for image classification. In: IEEE Congress on Evolutionary Computation (CEC), pp. 1–8 (2018). https://doi.org/10.1109/CEC.2018.8477735
Wang, B., Sun, Y., Xue, B., Zhang, M.: A hybrid differential evolution approach to designing deep convolutional neural networks for image classification. In: Mitrovic, T., Xue, B., Li, X. (eds.) AI 2018. LNCS (LNAI), vol. 11320, pp. 237–250. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-03991-2_24
Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)
Xie, L., Yuille, A.: Genetic CNN. In: IEEE International Conference on Computer Vision (ICCV), pp. 1379–1388 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Huang, J., Xue, B., Sun, Y., Zhang, M. (2022). EDE-NAS: An Eclectic Differential Evolution Approach to Single-Path Neural Architecture Search. In: Aziz, H., Corrêa, D., French, T. (eds) AI 2022: Advances in Artificial Intelligence. AI 2022. Lecture Notes in Computer Science(), vol 13728. Springer, Cham. https://doi.org/10.1007/978-3-031-22695-3_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-22695-3_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-22694-6
Online ISBN: 978-3-031-22695-3
eBook Packages: Computer ScienceComputer Science (R0)