Skip to main content
SpringerLink
Log in

Trust-Region Based Stochastic Variational Inference for Distributed and Asynchronous Networks

  • Published:
Journal of Systems Science and Complexity Aims and scope Submit manuscript

Abstract

Stochastic variational inference is an efficient Bayesian inference technology for massive datasets, which approximates posteriors by using noisy gradient estimates. Traditional stochastic variational inference can only be performed in a centralized manner, which limits its applications in a wide range of situations where data is possessed by multiple nodes. Therefore, this paper develops a novel trust-region based stochastic variational inference algorithm for a general class of conjugate-exponential models over distributed and asynchronous networks, where the global parameters are diffused over the network by using the Metropolis rule and the local parameters are updated by using the trust-region method. Besides, a simple rule is introduced to balance the transmission frequencies between neighboring nodes such that the proposed distributed algorithm can be performed in an asynchronous manner. The utility of the proposed algorithm is tested by fitting the Bernoulli model and the Gaussian model to different datasets on a synthetic network, and experimental results demonstrate its effectiveness and advantages over existing works.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Blei D M, Ng A Y, and Jordan M I, Latent dirichlet allocation, Journal of Machine Learning Research, 2003, 3: 993–1022.

    MATH  Google Scholar 

  2. Corduneanu A and Bishop C M, Variational bayesian model selection for mixture distributions, Artificial Intelligence and Statistics (Eds. by Jaakkola T and Richardson T), Morgan Kaufmann, Waltham, 2001.

    Google Scholar 

  3. Letham B, Letham L M, and Rudin C, Bayesian inference of arrival rate and substitution behavior from sales transaction data with stockouts, Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, USA, 2016.

  4. Hoffman M D, Blei D M, Wang C, et al., Stochastic variational inference, Journal of Machine Learning Research, 2013, 14(1): 1303–1347.

    MathSciNet  MATH  Google Scholar 

  5. Hu Y, Niu D, Yang J, et al., FDML: A collaborative machine learning framework for distributed features, Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Anchorage, USA, 2019.

  6. Liu S, Pan S J, and Ho Q, Distributed multi-task relationship learning, Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, Canada, 2017.

  7. Fang H, Liu S, and Wei Y, Distributed clustering algorithm for energy efficiency and load-balance in large-scale multi-agent systems, Journal of Systems Science and Complexity, 2018, 31(1): 234–243.

    Article  MATH  Google Scholar 

  8. Hua J and Li C, Distributed variational Bayesian algorithms over sensor networks, IEEE Transactions on Signal Processing, 2015, 64(3): 783–798.

    Article  MathSciNet  MATH  Google Scholar 

  9. Anwar H and Zhu Q, ADMM-based networked stochastic variational inference, arXiv preprint, arXiv: 1802.10168, 2018.

  10. Fu W, Qin J, and Zhu Y, Distributed stochastic variational inference based on diffusion method, Acta Automatica Sinica, 2021, 47(1): 92–99.

    MATH  Google Scholar 

  11. Mohamad S, Bouchachia A, and Sayed-Mouchaweh M, Asynchronous stochastic variational inference, Proceedings of INNS Big Data and Deep Learning Conference, Sestri Levante, Italy, 2019.

  12. Ye B, Qin J, Fu W, et al., Distributed bayesian inference over sensor networks, IEEE Transactions on Cybernetics, DOI: https://doi.org/10.1109/TCYB.2021.3106660, 2021.

  13. Zhang J, Raman P, Ji S, et al., Extreme stochastic variational inference: Distributed inference for large scale mixture models, International Conference on Artificial Intelligence and Statistics, Naha, Japan, 2019.

  14. Cattivelli F S and Sayed A H, Diffusion LMS strategies for distributed estimation, IEEE Transactions on Signal Processing, 2010, 58(3): 1035–1048.

    Article  MathSciNet  MATH  Google Scholar 

  15. Boyd S, Parikh N, and Chu E, Distributed Optimization and Statistical Learning Via the Alternating Direction Method of Multipliers, Now Publishers Inc, Hanover, USA, 2011.

    MATH  Google Scholar 

  16. García-Fernandez Á F and Grajal J, Asynchronous particle filter for tracking using non-synchronous sensor networks, Signal Processing, 2011, 91(10): 2304–2313.

    Article  Google Scholar 

  17. Niu F, Recht B, Re C, et al., HOGWILD!: A lock-free approach to parallelizing stochastic gradient descent, Proceedings of the 24th International Conference on Neural Information Processing Systems, Granada, Spain, 2011.

  18. Hughes M C and Sudderth E, Memoized online variational inference for Dirichlet process mixture models, Advances in Neural Information Processing Systems, 2013, 26: 1133–1141.

    Google Scholar 

  19. Hoffman M D and Blei D M, Structured stochastic variational inference, Proceedings of the 18th International Conference on Artificial Intelligence and Statistic, San Diego, USA, 2015.

  20. Nedic A, Olshevsky A, and Shi W, Achieving geometric convergence for distributed optimization over time-varying graphs, SIAM Journal on Optimization, 2017, 27(4): 2597–2633.

    Article  MathSciNet  MATH  Google Scholar 

  21. Nocedal J and Wright S, Numerical Optimization, Springer Science & Business Media, New York, USA, 2006.

    MATH  Google Scholar 

  22. Theis L and Hoffman M D, A trust-region method for stochastic variational inference with applications to streaming data, Proceedings of the 32nd International Conference on Machine Learning, Lille, France, 2015.

  23. Hoffman M and Blei D, Stochastic structured variational inference, Proceedings of the 18th International Conference on Artificial Intelligence and Statistics, San Diego, USA, 2015.

  24. Blei D M, Kucukelbir A, and McAuliffe J D, Variational inference: A review for statisticians, Journal of the American Statistical Association, 2017, 112(518): 859–877.

    Article  MathSciNet  Google Scholar 

  25. Betancourt M, The convergence of markov chain monte carlo methods: From the Metropolis method to Hamiltonian Monte Carlo, Annalen der Physik, 2019, 531(3): 1700214.

    Article  MathSciNet  Google Scholar 

  26. Durrett R, Probability: Theory and Examples, Cambridge University Press, New York, USA, 2019.

    Book  MATH  Google Scholar 

  27. Kingston D B and Beard R W, Discrete-time average-consensus under switching network topologies, Proceedings of the 2006 American Control Conference, Minneapolis, USA, 2006.

Download references

Author information

Authors and Affiliations

  1. Department of Automation, University of Science and Technology of China, Hefei, 230027, China

    Weiming Fu, Jiahu Qin, Yu Kang & Baijia Ye

  2. Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, 230088, China

    Jiahu Qin

  3. School of Computer Science and Engineering, and Guangdong Province Key Laboratory of Computational Science, Sun Yat-Sen University, Guangzhou, 510006, China

    Qing Ling

  4. Institute of Advanced Technology, University of Science and Technology of China, Hefei, 230027, China

    Yu Kang

Authors
  1. Weiming Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiahu Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Qing Ling
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yu Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Baijia Ye
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jiahu Qin.

Additional information

This research was supported in part by the National Natural Science Foundation of China under Grant Nos. 61922076, 61873252, 61725304, and 61973324; in part by Guangdong Basic and Applied Basic Research Foundation under Grant No. 2021B1515020094; and in part by the Guangdong Provincial Key Laboratory of Computational Science under Grant No. 2020B1212060032.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Fu, W., Qin, J., Ling, Q. et al. Trust-Region Based Stochastic Variational Inference for Distributed and Asynchronous Networks. J Syst Sci Complex 35, 2062–2076 (2022). https://doi.org/10.1007/s11424-022-2085-5

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11424-022-2085-5

Keywords

Search

Navigation