Abstract
Stochastic variational inference is an efficient Bayesian inference technology for massive datasets, which approximates posteriors by using noisy gradient estimates. Traditional stochastic variational inference can only be performed in a centralized manner, which limits its applications in a wide range of situations where data is possessed by multiple nodes. Therefore, this paper develops a novel trust-region based stochastic variational inference algorithm for a general class of conjugate-exponential models over distributed and asynchronous networks, where the global parameters are diffused over the network by using the Metropolis rule and the local parameters are updated by using the trust-region method. Besides, a simple rule is introduced to balance the transmission frequencies between neighboring nodes such that the proposed distributed algorithm can be performed in an asynchronous manner. The utility of the proposed algorithm is tested by fitting the Bernoulli model and the Gaussian model to different datasets on a synthetic network, and experimental results demonstrate its effectiveness and advantages over existing works.
Similar content being viewed by others
References
Blei D M, Ng A Y, and Jordan M I, Latent dirichlet allocation, Journal of Machine Learning Research, 2003, 3: 993–1022.
Corduneanu A and Bishop C M, Variational bayesian model selection for mixture distributions, Artificial Intelligence and Statistics (Eds. by Jaakkola T and Richardson T), Morgan Kaufmann, Waltham, 2001.
Letham B, Letham L M, and Rudin C, Bayesian inference of arrival rate and substitution behavior from sales transaction data with stockouts, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, USA, 2016.
Hoffman M D, Blei D M, Wang C, et al., Stochastic variational inference, Journal of Machine Learning Research, 2013, 14(1): 1303–1347.
Hu Y, Niu D, Yang J, et al., FDML: A collaborative machine learning framework for distributed features, Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Anchorage, USA, 2019.
Liu S, Pan S J, and Ho Q, Distributed multi-task relationship learning, Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, Canada, 2017.
Fang H, Liu S, and Wei Y, Distributed clustering algorithm for energy efficiency and load-balance in large-scale multi-agent systems, Journal of Systems Science and Complexity, 2018, 31(1): 234–243.
Hua J and Li C, Distributed variational Bayesian algorithms over sensor networks, IEEE Transactions on Signal Processing, 2015, 64(3): 783–798.
Anwar H and Zhu Q, ADMM-based networked stochastic variational inference, arXiv preprint, arXiv: 1802.10168, 2018.
Fu W, Qin J, and Zhu Y, Distributed stochastic variational inference based on diffusion method, Acta Automatica Sinica, 2021, 47(1): 92–99.
Mohamad S, Bouchachia A, and Sayed-Mouchaweh M, Asynchronous stochastic variational inference, Proceedings of INNS Big Data and Deep Learning Conference, Sestri Levante, Italy, 2019.
Ye B, Qin J, Fu W, et al., Distributed bayesian inference over sensor networks, IEEE Transactions on Cybernetics, DOI: https://doi.org/10.1109/TCYB.2021.3106660, 2021.
Zhang J, Raman P, Ji S, et al., Extreme stochastic variational inference: Distributed inference for large scale mixture models, International Conference on Artificial Intelligence and Statistics, Naha, Japan, 2019.
Cattivelli F S and Sayed A H, Diffusion LMS strategies for distributed estimation, IEEE Transactions on Signal Processing, 2010, 58(3): 1035–1048.
Boyd S, Parikh N, and Chu E, Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers, Now Publishers Inc, Hanover, USA, 2011.
García-Fernandez Á F and Grajal J, Asynchronous particle filter for tracking using non-synchronous sensor networks, Signal Processing, 2011, 91(10): 2304–2313.
Niu F, Recht B, Re C, et al., HOGWILD!: A lock-free approach to parallelizing stochastic gradient descent, Proceedings of the 24th International Conference on Neural Information Processing Systems, Granada, Spain, 2011.
Hughes M C and Sudderth E, Memoized online variational inference for Dirichlet process mixture models, Advances in Neural Information Processing Systems, 2013, 26: 1133–1141.
Hoffman M D and Blei D M, Structured stochastic variational inference, Proceedings of the 18th International Conference on Artificial Intelligence and Statistic, San Diego, USA, 2015.
Nedic A, Olshevsky A, and Shi W, Achieving geometric convergence for distributed optimization over time-varying graphs, SIAM Journal on Optimization, 2017, 27(4): 2597–2633.
Nocedal J and Wright S, Numerical Optimization, Springer Science & Business Media, New York, USA, 2006.
Theis L and Hoffman M D, A trust-region method for stochastic variational inference with applications to streaming data, Proceedings of the 32nd International Conference on Machine Learning, Lille, France, 2015.
Hoffman M and Blei D, Stochastic structured variational inference, Proceedings of the 18th International Conference on Artificial Intelligence and Statistics, San Diego, USA, 2015.
Blei D M, Kucukelbir A, and McAuliffe J D, Variational inference: A review for statisticians, Journal of the American Statistical Association, 2017, 112(518): 859–877.
Betancourt M, The convergence of markov chain monte carlo methods: From the Metropolis method to Hamiltonian Monte Carlo, Annalen der Physik, 2019, 531(3): 1700214.
Durrett R, Probability: Theory and Examples, Cambridge University Press, New York, USA, 2019.
Kingston D B and Beard R W, Discrete-time average-consensus under switching network topologies, Proceedings of the 2006 American Control Conference, Minneapolis, USA, 2006.
Author information
Authors and Affiliations
Corresponding author
Additional information
This research was supported in part by the National Natural Science Foundation of China under Grant Nos. 61922076, 61873252, 61725304, and 61973324; in part by Guangdong Basic and Applied Basic Research Foundation under Grant No. 2021B1515020094; and in part by the Guangdong Provincial Key Laboratory of Computational Science under Grant No. 2020B1212060032.
Rights and permissions
About this article
Cite this article
Fu, W., Qin, J., Ling, Q. et al. Trust-Region Based Stochastic Variational Inference for Distributed and Asynchronous Networks. J Syst Sci Complex 35, 2062–2076 (2022). https://doi.org/10.1007/s11424-022-2085-5
Received:
Revised:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11424-022-2085-5