Abstract
Deep Gaussian processes (DGPs) are the natural extension of Gaussian processes (GPs) to a multi-layer architecture. DGPs are powerful probabilistic models that have shown better results than standard GPs in terms of generalization performance and prediction uncertainty estimation. Nevertheless, exact inference in DGPs is intractable, making these models hard to train. For this task, current approaches in the literature use approximate inference techniques such as variational inference or approximate expectation propagation. In this work, we present a new method for inference in DGPs using an approximate inference technique based on Monte Carlo methods and the expectation propagation algorithm. Our experiments show that our method scales well to large datasets and that its performance is comparable or better than other state of the art methods. Furthermore, our training method leads to interesting properties in the predictive distribution of the DGP. More precisely, it is able to capture output noise that is dependent on the input and it can generate multimodal predictive distributions. These two properties, which are not shared by other state-of-the-art approximate inference methods for DGPs, are analyzed in detail in our experiments.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Available at https://github.com/Gonzalo933/dgp-aepmcm.
References
Abadi, M., et al.: TensorFlow: Large-scale machine learning on heterogeneous systems (2015)
Bui, T.D., Hernández-Lobato, J.M., Hernández-Lobato, D., Li, Y., Turner, R.E.: Deep Gaussian processes for regression using approximate expectation propagation. In: International Conference on International Conference on Machine Learning, pp. 1472–1481 (2016)
Calandra, R., Peters, J., Rasmussen, C.E., Deisenroth, M.P.: Manifold Gaussian processes for regression. In: International Joint Conference on Neural Networks, pp. 3338–3345 (2016)
Cutajar, K., Bonilla, E.V., Michiardi, P., Filippone, M.: Random feature expansions for deep Gaussian processes. In: International Conference on Machine Learning, pp. 884–893 (2017)
Damianou, A., Lawrence, N.: Deep Gaussian processes. In: International Conference on Artificial Intelligence and Statistics, pp. 207–215 (2013)
Depeweg, S., Hernández-Lobato, J.M., Doshi-Velez, F., Udluft, S.: Learning and policy search in stochastic dynamical systems with Bayesian neural networks. CoRR abs/1605.07127 (2016)
Dua, D., Graff, C.: UCI repository (2017). http://archive.ics.uci.edu/ml
Duvenaud, D., Lloyd, J., Grosse, R., Tenenbaum, J., Zoubin, G.: Structure discovery in nonparametric regression through compositional kernel search. In: Proceedings of the 30th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 28, pp. 1166–1174. PMLR (17–19 June 2013)
Havasi, M., Hernández-Lobato, J.M., Murillo-Fuentes, J.J.: Deep Gaussian processes with decoupled inducing inputs. In: arXiv:1801.02939 [stat.ML] (2018)
Havasi, M., Hernández-Lobato, J.M., Murillo-Fuentes, J.J.: Inference in deep Gaussian processes using stochastic gradient Hamiltonian Monte Carlo. In: Advances in Neural Information Processing Systems, pp. 7517–7527 (2018)
Hensman, J., Fusi, N., Lawrence, N.D.: Gaussian processes for big data. In: Uncertainty in Artificial Intellegence, pp. 282–290 (2013)
Heskes, T., Zoeter, O.: Expectation propagation for approximate inference in dynamic Bayesian networks. In: Uncertainty in Artificial Intelligence, pp. 216–223 (2002)
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006)
Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations, pp. 1–15 (2015)
Ko, J., Fox, D.: GP-BayesFilters: Bayesian filtering using Gaussian process prediction and observation models. Auton. Robots 27, 75–90 (2009)
Li, Y., Hernandez-Lobato, J.M., Turner, R.E.: Stochastic expectation propagation. In: Neural Information Processing Systems, pp. 2323–2331 (2015)
Pope, C.A., Gosling, J.P., Barber, S., Johnson, J., Yamaguchi, T., Feingold, G., Blackwell, P.: Modelling spatial heterogeneity and discontinuities using voronoi tessellations. ArXiv e-prints (2018)
Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. The MIT Press, Cambridge (2006)
Salimbeni, H., Deisenroth, M.: Doubly stochastic variational inference for deep Gaussian processes. In: Advances in Neural Information Processing Systems, pp. 4588–4599 (2017)
Salimbeni, H., Dutordoir, V., Hensman, J., Deisenroth, M.: Deep Gaussian processes with importance-weighted variational inference. In: International Conference on Machine Learning, pp. 5589–5598 (2019)
Seeger, M.: Expectation propagation for exponential families. Technical report (2005)
Snelson, E., Ghahramani,: Sparse Gaussian processes using pseudo-inputs. In: Advances in Neural Information Processing Systems, pp. 1257–1264 (2006)
Snoek, J., Larochelle, H., Adams, R.P.: Practical Bayesian optimization of machine learning algorithms. In: Advances in Neural Information Processing Systems, pp. 2951–2959 (2012)
Titsias, M.: Variational learning of inducing variables in sparse Gaussian processes. In: International Conference on Artificial Intelligence and Statistics, pp. 567–574 (2009)
Vafa, K.: Training and inference for deep Gaussian processes. Technical report (last year project), Harvard College (2016)
Wilson, A.G., Adams, R.P.: Gaussian process kernels for pattern discovery and extrapolation. In: International Conference on International Conference on Machine Learning, pp. 1067–1075 (2013)
Wilson, A.G., Hu, Z., Salakhutdinov, R., Xing, E.P.: Deep kernel learning. In: International Conference on Artificial Intelligence and Statistics, pp. 370–378 (2016)
Yu, H., Chen, Y., Low, B.K.H., Jaillet, P., Dai, Z.: Implicit posterior variational inference for deep Gaussian processes. Adv. Neural Inf. Process. Syst. 32, 14502–14513 (2019)
Acknowledgment
We acknowledge using the facilities of Centro de Computación Científica at UAM and support from the Spanish Plan Nacional I+D+i (grants TIN2016-76406-P, TEC2016-81900-REDT and PID2019-106827GB-I00) and from Comunidad de Madrid (grants PEJD-2016-TIC-238 and PEJD-2017-PRE-TIC-3776).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Hernández-Muñoz, G., Villacampa-Calvo, C., Hernández-Lobato, D. (2021). Deep Gaussian Processes Using Expectation Propagation and Monte Carlo Methods. In: Hutter, F., Kersting, K., Lijffijt, J., Valera, I. (eds) Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2020. Lecture Notes in Computer Science(), vol 12459. Springer, Cham. https://doi.org/10.1007/978-3-030-67664-3_29
Download citation
DOI: https://doi.org/10.1007/978-3-030-67664-3_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-67663-6
Online ISBN: 978-3-030-67664-3
eBook Packages: Computer ScienceComputer Science (R0)
