Abstract
Mammalian brains exhibit functional self-organisation between different neocortical regions to form virtual hierarchies from a physical 2D sheet. We propose a biologically-inspired self-organizing neural network architecture emulating the same. The network is composed of autoencoder units and driven by a meta-learning rule based on maximizing the Shannon entropy of latent representations of the input, which optimizes the receptive field placement of each unit within a feature map. Unlike Neural Architecture Search, here both the network parameters and the architecture are learned simultaneously. In a case study on image datasets, we observe that the meta-learning rule causes a functional hierarchy to form, and leads to learning progressively better topological configurations and higher classification performance overall, starting from randomly initialized architectures. In particular, our approach yields competitive performance in terms of classification accuracy compared to optimal handcrafted architecture(s) with desirable topological features for this network type, on both MNIST and CIFAR-10 datasets, even though it is not as significant for the latter.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The code is available on github under: https://github.com/Cerenaut/self-organizing.
- 2.
References
Chen, L.C., et al.: Searching for efficient multi-scale architectures for dense image prediction. In: Advances in Neural Information Processing Systems, pp. 8699–8710 (2018)
Corominas-Murtra, B., Goñi, J., Solé, R.V., RodrÃguez-Caso, C.: On the origins of hierarchy in complex networks. Proc. Natl. Acad. Sci. 110(33), 13316–13321 (2013)
Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: a survey. arXiv preprint arXiv:1808.05377 (2018)
Friston, K.: A free energy principle for biological systems. Entropy 14(11), 2100–2121 (2012)
Goldberg, E.: Gradiental approach to neocortical functional organization. J. Clin. Exp. Neuropsychol. 11(4), 489–517 (1989)
Huang, G., Sun, Yu., Liu, Z., Sedra, D., Weinberger, K.Q.: Deep networks with stochastic depth. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 646–661. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_39
Kandel, E.R., et al.: Principles of Neural Science, vol. 4. McGraw-Hill, New York (2000)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
Larsson, G., Maire, M., Shakhnarovich, G.: FractalNet: ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648 (2016)
Lee, C.Y., Xie, S., Gallagher, P., Zhang, Z., Tu, Z.: Deeply-supervised nets. In: Artificial Intelligence and Statistics, pp. 562–570 (2015)
Lin, M., Chen, Q., Yan, S.: Network in network. arXiv preprint arXiv:1312.4400 (2013)
Lin, Z., Memisevic, R., Konda, K.: How far can we go without convolution: improving fully-connected networks. arXiv preprint arXiv:1511.02580 (2015)
Purwani, S., Nahar, J., Twining, C.: Analyzing bin-width effect on the computed entropy. In: AIP Conference Proceedings, vol. 1868, p. 040008. AIP Publishing LLC (2017)
Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: FitNets: hints for thin deep nets. arXiv preprint arXiv:1412.6550 (2014)
Russell, S.J., Norvig, P.: Artificial Intelligence: A Modern Approach. Pearson Education Limited, Kuala Lumpur (2016)
Saxe, A.M., et al.: On the information bottleneck theory of deep learning. J. Stat. Mech. Theory Exp. 2019(12), 124020 (2019)
Shwartz-Ziv, R., Tishby, N.: Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810 (2017)
Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.: Striving for simplicity: the all convolutional net. arXiv preprint arXiv:1412.6806 (2014)
Srivastava, R.K., Greff, K., Schmidhuber, J.: Highway networks. arXiv preprint arXiv:1505.00387 (2015)
Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99–127 (2002)
Tishby, N., Zaslavsky, N.: Deep learning and the information bottleneck principle. In: 2015 IEEE Information Theory Workshop (ITW), pp. 1–5. IEEE (2015)
Zoph, B., Le, Q.V.: Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 (2016)
Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: Proceedings of the IEEE Conference on Computer Vision And Pattern Recognition, pp. 8697–8710 (2018)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Borgohain, S. et al. (2020). Self-organising Neural Network Hierarchy. In: Gallagher, M., Moustafa, N., Lakshika, E. (eds) AI 2020: Advances in Artificial Intelligence. AI 2020. Lecture Notes in Computer Science(), vol 12576. Springer, Cham. https://doi.org/10.1007/978-3-030-64984-5_28
Download citation
DOI: https://doi.org/10.1007/978-3-030-64984-5_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-64983-8
Online ISBN: 978-3-030-64984-5
eBook Packages: Computer ScienceComputer Science (R0)