Abstract
In deep learning research, typical neural network models are multi-layered architectures, and weights are tuned while optimizing a carefully designed loss function. In recent years, studies of randomized neural networks have been extended towards deep architectures, opening a new research direction to the design of deep learning models. However, how the structure of the network can influence the model performance still remains unclear. In this paper, we move a further step to investigate the relation between network topology and performance via a structure evolution algorithm. Experimental results show that the graph would evolve towards a more small-world topology at the beginning of the training session along with gaining accuracy, and would also evolve towards a structure with more scale-free property in the following periods. These conclusions could help explain the effectiveness of the randomly connected networks, as well as give us insights in new possibilities of network architecture design.
This research is partially supported by the NSF of China (No. 61876197) and the Beijing NSF (Grant No. 7192105).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Fodor, J.A., Pylyshyn, Z.W.: Connectionism and cognitive architecture: a critical analysis. Cognition 28, 3–71 (1988)
Turing, A.: Intelligent machinery. The Essential Turing, pp. 395–432 (1948)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, vol. 25, no. 1097–1105 (2012)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Conference Track Proceedings of ICLR (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of CVPR, vol. 2016, no. 770–778 (2016)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of CVPR, vol. 2017, no. 4700–4708 (2017)
Xie, S., Kirillov, A., Girshick, R., He, K.: Exploring randomly wired neural networks for image recognition. In: Proceedings of ICCV, vol. 2019, no. 1284–1293 (2019)
Javaheripi, M., Rouhani, B.D., Koushanfar, F.: SWNet: small-world neural networks and rapid convergence. arXiv preprint arXiv:1904.04862 (2019)
Teuscher, C., Sanchez, E.: A revival of Turing’s forgotten connectionist ideas: exploring unorganized machines. In: French, R.M., Sougné, J.P. (eds.) Connectionist Models of Learning, Development and Evolution, pp. 153–162 (2001). Springer, London. https://doi.org/10.1007/978-1-4471-0281-6_16
Wortsman, M., Farhadi, A., Rastegari, M.: Discovering neural wirings. In: Advances in Neural Information Processing Systems, vol. 32, no. 2680–2690 (2019)
Rosenfeld, A., Tsotsos, J.K.: Intriguing properties of randomly weighted networks: generalizing while learning next to nothing. In: Proceedings of CRV, vol. 2019, no. 9–16 (2019)
Frankle, J., Carbin, M.: The lottery ticket hypothesis: finding sparse, trainable neural networks. In: Proceedings of ICLR (2019)
Zhou, H., Lan, J., Liu, R., Yosinski, J.: Deconstructing lottery tickets: zeros, signs, and the supermask. In: Advances in Neural Information Processing Systems, vol. 32, pp. 3592–3602 (2019)
Ramanujan, V., Wortsman, M., Kembhavi, A., Farhadi, A., Rastegari, M.: What’s hidden in a randomly weighted neural network? In: Proceedings of CVPR, vol. 2020, no. 11893–11902 (2020)
Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report, University of Toronto (2009)
You, J., Leskovec, J., He, K., Xie, S.: Graph structure of neural networks. In: Proceedings of the ICML, vol. 2020, no. 10881–10891 (2020)
Nair, V., Hinton, G.E.: Rectified linear units improve restricted Boltzmann machines. In: Proceedings of ICML, vol. 2010, no. 807–814 (2010)
Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: the missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on ImageNet classification. In: Proceedings of ICCV, vol. 2015, no. 1026–1034 (2015)
Erdős, P., Rényi, A.: On the evolution of random graphs. Publ. Math. Inst. Hung. Acad. Sci. 5, 17–60 (1960)
Bassett, D.S., Bullmore, E.T.: Small-world brain networks revisited. Neuroscientist 23, 499–516 (2017)
Onnela, J.-P., Saramäki, J., Kertész, J., Kaski, K.: Intensity and coherence of motifs in weighted complex networks. Phys. Rev. E 71, 065103 (2005)
Barabási, A.-L., Bonabeau, E.: Scale-free networks. Sci. Am. 288, 60–69 (2003)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: Proceedings of ICML, vol. 2015, no. 448–456 (2015)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhou, Y., He, Z., Wan, T., Qin, Z. (2021). Random Neural Graph Generation with Structure Evolution. In: Mantoro, T., Lee, M., Ayu, M.A., Wong, K.W., Hidayanto, A.N. (eds) Neural Information Processing. ICONIP 2021. Lecture Notes in Computer Science(), vol 13109. Springer, Cham. https://doi.org/10.1007/978-3-030-92270-2_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-92270-2_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-92269-6
Online ISBN: 978-3-030-92270-2
eBook Packages: Computer ScienceComputer Science (R0)