Abstract
It has been recently shown that a deep neural network with i.i.d. random parameters is equivalent to a Gaussian process in the limit of infinite network width. The Gaussian process associated to the neural network is fully described by a recursive covariance kernel determined by the architecture of the network, and which is expressed in terms of expectation. We give a numerically workable analytic expression of the neural network recursive covariance based on Hermite polynomials. We give explicit forms of this recursive covariance for the cases of neural networks with activation function the Heaviside, ReLU and sigmoid.
A. Arratia—Supported by grant TIN2017-89244-R from MINECO (Ministerio de Economía, Industria y Competitividad) and the recognition 2017SGR-856 (MACDA) from AGAUR (Generalitat de Catalunya).
A. Cabaña—Supported by grant RTI2018-096072-B-I00D Ministerio de Ciencia, Innovación y Universidades.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. The MIT Press, Cambridge (2016)
Blum, J.R., Chernoff, H., Rosenblatt, M., Teicher, H.: Central limit theorems for interchangeable processes. Canad. J. Math. 10, 222–229 (1958)
Eddelbuettel, D., Romain Francois, R.: RCPP: Seamless R and C++ Integration. J. Stat. Softw. 40(8), 1–18 (2011)
Kallenberg, O.: Probabilistic Symmetries and Invariance Principles. Springer Series Probability and Applications. Springer, Heidelberg (2005). https://doi.org/10.1007/0-387-28861-9
Karatzoglou, A., Smola, A., Hornik, K., Zeileis, A.: kernlab - an S4 package for kernel methods in R. J. Stat. Softw. 11(9), 1–20 (2004)
Lee, J., Bahri, Y., Novak, R., Schoenholz, S.S., Pennington, J., Sohl-Dickstein, J.: Deep neural networks as Gaussian processes. In: International Conference on Learning Representations, vol. 4 (2018)
de G. Matthews, A.G., Hron, J., Rowland, M., Turner, R.E., Ghahramani, Z.: Gaussian process behaviour in wide deep neural networks. In: International Conference on Learning Representations, vol. 4 (2018)
de G. Matthews, A.G., Hron, J., Rowland, M., Turner, R.E., Ghahramani, Z.: Gaussian process behaviour in wide deep neural networks. arXiv:1804.11271 (2018)
Neal, R.M.: Bayesian learning for neural networks. Ph.D. Thesis, Department of Computer Science, University of Toronto (1994)
Peccati, G., Taqqu, M.: Wiener Chaos, Moments, Cumulants and Diagrams. Bocconi & Springer (2011)
Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning, vol. 1. MIT Press, Cambridge (2006)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Arratia, A., Cabaña, A., León, J.R. (2020). Deep and Wide Neural Networks Covariance Estimation. In: Farkaš, I., Masulli, P., Wermter, S. (eds) Artificial Neural Networks and Machine Learning – ICANN 2020. ICANN 2020. Lecture Notes in Computer Science(), vol 12396. Springer, Cham. https://doi.org/10.1007/978-3-030-61609-0_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-61609-0_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-61608-3
Online ISBN: 978-3-030-61609-0
eBook Packages: Computer ScienceComputer Science (R0)