Skip to main content
SpringerLink
Log in

Evolving graph convolutional networks for neural architecture search

  • Special issue on Advances of Neural Computing phasing challenges in the era of 4th industrial revolution
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

As neural architecture search (NAS) becomes an increasingly adopted method to design network architectures, various methods have been proposed to speedup the process. Besides proxy evaluation tasks, weight sharing, and scaling down the evaluated architectures, performance-predicting models exhibit multiple advantages. Eliminating the need to train candidate architectures and enabling transfer learning between datasets, researchers can also utilize them as a surrogate function for Bayesian optimization. On the other hand, graph convolutional networks (GCNs) have also been increasingly adopted for various tasks, enabling deep learning techniques on graphs without feature engineering. In this paper, we employ an evolutionary-based NAS method to evolve GCNs for the problem of predicting the relative performance of various architectures included in the NAS-Bench-101 dataset. By fine-tuning the architecture generated by our methodology, we manage to achieve a Kendall’s tau correlation coefficient of 0.907 between 1050 completely unseen architectures, utilizing only 450 samples, while also outperforming a strong baseline on the same task. Furthermore, we validate our method on custom global search space architectures, generated for the Fashion-MNIST dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Cai H, Yang J, Zhang W, Han S, Yu Y (2018) Path-level network transformation for efficient architecture search. arXiv:1806.02639

  2. Chau T, Dudziak Ł, Abdelfattah MS, Lee R, Kim H, Lane ND (2020) Brp-nas: prediction-based nas using gcns. arXiv:2007.08668

  3. Deng B, Yan J, Lin D (2017) Peephole: predicting network performance before training. arXiv:1712.03351

  4. Dong X, Yang Y (2020) Nas-bench-102: Extending the scope of reproducible neural architecture search. arXiv:2001.00326

  5. Gao Y, Yang H, Zhang P, Zhou C, Hu Y (2019) Graphnas: graph neural architecture search with reinforcement learning. arXiv:1904.09981

  6. Hamilton W, Ying Z, Leskovec J (2017) Inductive representation learning on large graphs

  7. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  8. Kacprzyk J, Pedrycz W (2015) Springer handbook of computational intelligence. Springer, Berlin

    Book  Google Scholar 

  9. Kandasamy K, Neiswanger W, Schneider J, Poczos B, Xing EP (2018) Neural architecture search with Bayesian optimisation and optimal transport. arXiv:1802.07191

  10. Kendall MG (1938) A new measure of rank correlation. Biometrika 30(1/2):81–93

    Article  Google Scholar 

  11. Kipf TN, Welling M (2016) Semi-supervised classification with graph convolutional networks. arXiv:1609.02907

  12. Krizhevsky A, Hinton G et al. (2009) Learning multiple layers of features from tiny images

  13. Kyriakides G, Margaritis K (2020) The effect of reduced training in neural architecture search. Neural Comput Appl 32(23):17321–17332

    Article  Google Scholar 

  14. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  15. Liu C, Zoph B, Neumann M, Shlens J, Hua W, Li LJ, Fei-Fei L, Yuille A, Huang J, Murphy K (2018) Progressive neural architecture search. http://github.com/tensorflow/

  16. Liu H, Simonyan K, Vinyals O, Fernando C, Kavukcuoglu K (2018) Hierarchical representations for efficient architecture search. In: 6th international conference on learning representations, ICLR 2018—conference track proceedings. arxiv:1711.00436

  17. Long D, Zhang S, Zhang Y (2019) Performance prediction based on neural architecture features. Cogn Comput Syst 2(2):80–83

    Article  Google Scholar 

  18. Loshchilov I, Hutter F (2017) Decoupled weight decay regularization. arXiv:1711.05101

  19. Luo R, Tian F, Qin T, Chen E, Liu TY (2018) Neural architecture optimization. Adv Neural Inf Process Syst 2018:7816–7827

    Google Scholar 

  20. Miikkulainen R, Liang J, Meyerson E, Rawal A, Fink D, Francon O, Raju B, Shahrzad H, Navruzyan A, Duffy N et al (2019) Evolving deep neural networks. In: Artificial intelligence in the age of neural networks and brain computing. Elsevier, pp 293–312

  21. Pham H, Guan MY, Zoph B, Le QV, Dean J (2018) Efficient neural architecture search via parameter sharing. In: 35th international conference on machine learning, ICML 2018, vol 9, pp. 6522–6531. arXiv:1802.03268

  22. Real E, Aggarwal A, Huang Y, Le QV (2019) Regularized evolution for image classifier architecture search. Proc AAAI Conf Artif Intell 33:4780–4789. https://doi.org/10.1609/aaai.v33i01.33014780

    Article  Google Scholar 

  23. Schlichtkrull M, Kipf TN, Bloem P, Van Den Berg R, Titov I, Welling M (2018) Modeling relational data with graph convolutional networks

  24. Smith LN (2017) Cyclical learning rates for training neural networks. In: 2017 IEEE winter conference on applications of computer vision (WACV). IEEE, pp 464–472

  25. Stanley KO, Miikkulainen R (2002) Evolving neural networks through augmenting topologies. Evol Comput 10(2):99–127

    Article  Google Scholar 

  26. Sun Y, Wang H, Xue B, Jin Y, Yen GG, Zhang M (2019) Surrogate-assisted evolutionary deep learning using an end-to-end random forest-based performance predictor. IEEE Trans Evol Comput 24(2):350–364

    Article  Google Scholar 

  27. Tang Y, Wang Y, Xu Y, Chen H, Shi B, Xu C, Xu C, Tian Q, Xu C (2020) A semi-supervised assessor of neural architectures. In: IEEE/CVF conference on computer vision and pattern recognition, pp. 1810–1819

  28. Xiao H, Rasul K, Vollgraf R (2017) Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747

  29. Xu Y, Wang Y, Han K, Jui S, Xu C, Tian Q, Xu C (2019) Renas: relativistic evaluation of neural architecture search. arXiv:1910.01523

  30. Ying C, Klein A, Real E, Christiansen E, Murphy K, Hutter F (2019) Nas-bench-101: towards reproducible neural architecture search. arXiv:1902.09635

  31. Zela A, Siems J, Hutter F (2020) Nas-bench-1shot1: benchmarking and dissecting one-shot neural architecture search. arXiv:2001.10422

  32. Zhao H, Wei L, Yao Q (2020) Simplifying architecture search for graph neural network. arXiv:2008.11652

  33. Zhong Z, Yan J, Wu W, Shao J, Liu CL (2018) Practical block-wise neural network architecture generation. In: IEEE conference on computer vision and pattern recognition, pp. 2423–2432

  34. Zhou K, Song Q, Huang X, Hu X (2019) Auto-gnn: neural architecture search of graph neural networks. arXiv:1909.03184

  35. Zoph B, Le QV (2016) Neural architecture search with reinforcement learning. arXiv:1611.01578

  36. Zoph B, Vasudevan V, Shlens J, Le QV (2018) Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition, pp 8697–8710. arXiv:1707.07012

Download references

Acknowledgements

This work was supported by computational time granted from the Greek Research & Technology Network (GRNET) in the National HPC facility—ARIS—under Project ID ECN, which enabled the development of the method. The research work was supported by the Hellenic Foundation for Research and Innovation (HFRI) under the HFRI Ph.D. Fellowship Grant (Fellowship Number: 646).

Author information

Authors and Affiliations

  1. Department of Applied Informatics, University of Macedonia, Thessaloniki, 55236, Greece

    George Kyriakides & Konstantinos Margaritis

Authors
  1. George Kyriakides
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Konstantinos Margaritis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to George Kyriakides.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Funded by the Hellenic Foundation for Research and Innovation (HFRI) under the HFRI PhD Fellowship Grant (Fellowship Number: 646).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kyriakides, G., Margaritis, K. Evolving graph convolutional networks for neural architecture search. Neural Comput & Applic 34, 899–909 (2022). https://doi.org/10.1007/s00521-021-05979-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-05979-8

Keywords

Search

Navigation