Skip to main content
SpringerLink
Log in

Multi-objective Evolution for Deep Neural Network Architecture Search

  • Conference paper
  • First Online:
Neural Information Processing (ICONIP 2020)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12534))

Included in the following conference series:

Abstract

In this paper, we propose a multi-objective evolutionary algorithm for automatic deep neural architecture search. The algorithm optimizes the performance of the model together with the number of network parameters. This allows exploring architectures that are both successful and compact. We test the proposed solution on several image classification data sets including MNIST, fashionMNIST and CIFAR-10, and we consider deep architectures including convolutional and fully connected networks. The effects of using two different versions of multi-objective selections are also examined in the paper. Our approach outperforms both the considered baseline architectures and the standard genetic algorithm used in our previous work.

This work was partially supported by the Czech Science Foundation project no. 18-23827S and institutional support of the Institute of Computer Science RVO 67985807.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bergstra, J., Yamins, D., Cox, D.D.: Making a science of model search: hyperparameter optimization in hundreds of dimensions for vision architectures. In: Proceedings of the 30th International Conference on Machine Learning (ICML 2013) (2013)

    Google Scholar 

  2. Chollet, F.: Keras (2015). https://github.com/fchollet/keras

  3. Deb, K., Jain, H.: An evolutionary many-objective optimization algorithm using reference-points-based nondominated sorting approach. IEEE Trans. Evol. Comput. 18(4), 577–601 (2014)

    Article  Google Scholar 

  4. Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002). https://doi.org/10.1109/4235.996017

    Article  Google Scholar 

  5. Elsken, T., Metzen, J.H., Hutter, F.: Efficient multi-objective neural architecture search via Lamarckian evolution. In: International Conference on Learning Representations (2019). https://openreview.net/forum?id=ByME42AqK7

  6. Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: a survey. J. Mach. Learn. Res. 20(55), 1–21 (2019). http://jmlr.org/papers/v20/18-598.html

    MathSciNet  MATH  Google Scholar 

  7. Gong, C., Jiang, Z., Wang, D., Lin, Y., Liu, Q., Pan, D.Z.: Mixed precision neural architecture search for energy efficient deep learning. In: 2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp. 1–7 (2019)

    Google Scholar 

  8. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016). http://www.deeplearningbook.org

  9. Guo, Y., et al.: NAT: neural architecture transformer for accurate and compact architectures. In: NeurIPS (2019)

    Google Scholar 

  10. Goodfellow, I., et al.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). Software available from tensorflow.org. https://www.tensorflow.org/

  11. Jin, H., Song, Q., Hu, X.: Auto-Keras: an efficient neural architecture search system. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1946–1956. ACM (2019)

    Google Scholar 

  12. Keras + Hyperopt: A very simple wrapper for convenient hyperparameter optimization. http://maxpumperla.com/hyperas/

  13. Krizhevsky, A., Nair, V., Hinton, G.: CIFAR-10 (Canadian Institute for Advanced Research). http://www.cs.toronto.edu/kriz/cifar.html

  14. Lecun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015). https://doi.org/10.1038/nature14539

    Article  Google Scholar 

  15. Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998). https://doi.org/10.1109/5.726791

    Article  Google Scholar 

  16. LeCun, Y., Cortes, C.: The MNIST database of handwritten digits (2012). http://research.microsoft.com/apps/pubs/default.aspx?id=204699

  17. Lu, Z., Deb, K., Boddeti, V.: MUXConv: information multiplexing in convolutional neural networks, pp. 12041–12050, June 2020. https://doi.org/10.1109/CVPR42600.2020.01206

  18. Miikkulainen, R., et al.: Evolving deep neural networks. CoRR abs/1703.00548 (2017). http://arxiv.org/abs/1703.00548

  19. Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002). http://nn.cs.utexas.edu/?stanley:ec02

    Article  Google Scholar 

  20. Autonomio Talos [computer software] (2019). http://github.com/autonomio/talos

  21. Vidnerová, P., Neruda, R.: Evolution strategies for deep neural network models design. In: Hlaváčová, J. (ed.) Proceedings ITAT 2017: Information Technologies - Applications and Theory. CEUR Workshop Proceedings, vol. 1885, pp. 159–166. Technical University & CreateSpace Independent Publishing Platform, Aachen & Charleston (2017)

    Google Scholar 

  22. Vidnerová, P., Neruda, R.: Evolving Keras architectures for sensor data analysis. In: 2017 Federated Conference on Computer Science and Information Systems (FedCSIS), pp. 109–112, September 2017. https://doi.org/10.15439/2017F241

  23. Vidnerová, P., Procházka, Š.: NSGA-Keras: Neural architecture search for Keras sequential models (2020). https://github.com/PetraVidnerova/nsga-keras

  24. Vidnerová, P., Neruda, R.: Asynchronous evolution of convolutional networks. In: Krajci, S. (ed.) Proceedings of the 18th Conference Information Technologies - Applications and Theory (ITAT 2018), Hotel Plejsy, Slovakia, 21–25 September 2018. CEUR Workshop Proceedings, vol. 2203, pp. 80–85. CEUR-WS.org (2018). http://ceur-ws.org/Vol-2203/80.pdf

  25. Xia, W., Yin, H., Jha, N.: Efficient synthesis of compact deep neural networks. CoRR abs/2004.08704 (2020). http://arxiv.org/abs/2004.08704

  26. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. The Czech Academy of Sciences, Institute of Computer Science, Pod Vodárenskou Věží 2, 182 07, Prague 8, Czech Republic

    Petra Vidnerová & Roman Neruda

Authors
  1. Petra Vidnerová
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Roman Neruda
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Petra Vidnerová .

Editor information

Editors and Affiliations

  1. Department of AI, Ping An Life, Shenzhen, China

    Haiqin Yang

  2. Faculty of Information Technology, King Mongkut's Institute of Technology Ladkrabang, Bangkok, Thailand

    Kitsuchart Pasupa

  3. City University of Hong Kong, Kowloon, Hong Kong

    Andrew Chi-Sing Leung

  4. Department of Computer Science and Engineering, Hong Kong University of Science and Technology, Hong Kong, Hong Kong

    James T. Kwok

  5. School of Information Technology, King Mongkut’s University of Technology Thonburi, Bangkok, Thailand

    Jonathan H. Chan

  6. The Chinese University of Hong Kong, New Territories, Hong Kong

    Irwin King

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Vidnerová, P., Neruda, R. (2020). Multi-objective Evolution for Deep Neural Network Architecture Search. In: Yang, H., Pasupa, K., Leung, A.CS., Kwok, J.T., Chan, J.H., King, I. (eds) Neural Information Processing. ICONIP 2020. Lecture Notes in Computer Science(), vol 12534. Springer, Cham. https://doi.org/10.1007/978-3-030-63836-8_23

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-63836-8_23

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-63835-1

  • Online ISBN: 978-3-030-63836-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation