Abstract
The Dirichlet Belief Network (DirBN) has been proposed as a promising deep generative model that uses Dirichlet distributions to form layer-wise connections and thereby construct a multi-stochastic layered deep architecture. However, the DirBN cannot simultaneously achieve both sparsity, whereby the generated latent distributions place weights on a subset of components, and smoothness, which requires that the posterior distribution should not be dominated by the data. To address this limitation we introduce the sparse and smooth Dirichlet Belief Network (ssDirBN) which can achieve both sparsity and smoothness simultaneously, thereby increasing modelling flexibility over the DirBN. This gain is achieved by introducing binary variables to indicate whether each entity’s latent distribution at each layer uses a particular component. As a result, each latent distribution may use only a subset of components in each layer, and smoothness is enforced on this subset. Extra efforts on modifying the models are also made to fix the issues which is caused by introducing these binary variables. Extensive experimental results on real-world data show significant performance improvements of ssDirBN over state-of-the-art models in terms of both enhanced model predictions and reduced model complexity.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Airoldi, E.M., Blei, D.M., Fienberg, S.E., Xing, E.P.: Mixed membership stochastic block models. J. Mach. Learn. Res. 9, 1981–2014 (2008)
Burkhardt, S., Kramer, S.: Decoupling sparsity and smoothness in the Dirichlet variational autoencoder topic model. J. Mach. Learn. Res. 20(131), 1–27 (2019)
Dunson, D.B., Herring, A.H.: Bayesian latent variable models for mixed discrete outcomes. Biostatistics 6(1), 11–25 (2005)
Fan, X., Li, B., Li, C., Sisson, S., Chen, L.: Scalable deep generative relational model with high-order node dependence. In: NeurIPS, pp. 12637–12647 (2019)
Fan, X., Li, B., Li, Y., Sisson, S.: Poisson-randomised DirBN: large mutation is needed in dirichlet belief networks. In: ICML (2021)
Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In: ICML, pp. 1050–1059 (2016)
Guo, D., Chen, B., Lu, R., Zhou, M.: Recurrent hierarchical topic-guided RNN for language generation. In: ICML, pp. 3810–3821 (2020)
Guo, D., Chen, B., Zhang, H., Zhou, M.: Deep Poisson gamma dynamical systems. In: NeurIPS, pp. 8442–8452 (2018)
Hamilton, W., Ying, Z., Leskovec, J.: Inductive representation learning on large graphs. In: NIPS, pp. 1024–1034 (2017)
Hu, C., Rai, P., Carin, L.: Deep generative models for relational data with side information. In: ICML, pp. 1578–1586 (2017)
Kipf, T.N., Welling, M.: Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 (2016)
Li, Y., Fan, X., Chen, L., Li, B., Yu, Z., Sisson, S.A.: Recurrent Dirichlet belief networks for interpretable dynamic relational data modelling. In: IJCAI, pp. 2470–2476 (2020)
Schein, A., Linderman, S., Zhou, M., Blei, D., Wallach, H.: Poisson-randomized gamma dynamical systems. In: NeurIPS, pp. 782–793 (2019)
Schein, A., Wallach, H., Zhou, M.: Poisson-gamma dynamical systems. In: NIPS, pp. 5005–5013 (2016)
Sen, P., Namata, G., Bilgic, M., Getoor, L., Galligher, B., Eliassi-Rad, T.: Collective classification in network data. AI Mag. 29, 93 (2008)
Wang, C., Blei, D.: Decoupling sparsity and smoothness in the discrete hierarchical Dirichlet process. In: NIPS, pp. 1982–1989. Curran Associates Inc. (2009)
Yang, S., Koeppl, H.: A Poisson gamma probabilistic model for latent node-group memberships in dynamic networks. In: AAAI (2018)
Zhang, H., Chen, B., Guo, D., Zhou, M.: WHAI: Weibull hybrid autoencoding inference for deep topic modeling. In: International Conference on Learning Representations (2018)
Zhao, H., Du, L., Buntine, W.: Leveraging node attributes for incomplete relational data. In: ICML, pp. 4072–4081 (2017)
Zhao, H., Du, L., Buntine, W., Zhou, M.: Dirichlet belief networks for topic structure learning. In: NeurIPS, pp. 7955–7966 (2018)
Zhou, M.: Infinite edge partition models for overlapping community detection and link prediction. In: AISTATS, pp. 1135–1143 (2015)
Zhou, M., Cong, Y., Chen, B.: Augmentable gamma belief networks. J. Mach. Learn. Res. 17(163), 1–44 (2016)
Zitnik, M., Leskove, J.: Predicting multicellular function through multi-layer tissue networks. Bioinformatics 33, i190–i198 (2017)
Acknowledgements
Yaqiong Li is a recipient of UTS Research Excellence Scholarship. Xuhui Fan and Scott A. Sisson are supported by the Australian Research Council (ARC) through the Australian Centre of Excellence in Mathematical and Statistical Frontiers (ACEMS, CE140100049), and Scott A. Sisson through the ARC Future Fellow Scheme (FT170100079). Bin Li is supported in part by STCSM Project (20511100400), Shanghai Municipal Science and Technology Major Projects (2018SHZDZX01, 2021SHZDZX0103), and the Program for Professor of Special Appointment (Eastern Scholar) at Shanghai Institutions of Higher Learning.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Li, Y., Fan, X., Chen, L., Li, B., Sisson, S.A. (2021). Decoupling Sparsity and Smoothness in Dirichlet Belief Networks. In: Oliver, N., Pérez-Cruz, F., Kramer, S., Read, J., Lozano, J.A. (eds) Machine Learning and Knowledge Discovery in Databases. Research Track. ECML PKDD 2021. Lecture Notes in Computer Science(), vol 12976. Springer, Cham. https://doi.org/10.1007/978-3-030-86520-7_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-86520-7_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86519-1
Online ISBN: 978-3-030-86520-7
eBook Packages: Computer ScienceComputer Science (R0)
