Skip to main content

Advertisement

Springer Nature Link
Log in

Extracting nonlinear neural topics with neural variational bayes

  • Published:
World Wide Web Aims and scope Submit manuscript

Abstract

Recently, topic modeling has been upgraded by neural variational inference, which simultaneously allows the model structures deeper and proposes efficient update rules with the reparameterization trick. We formally call this recent new art as neural topic model. In this paper, we investigate a problem of neural topic models, where they formulate topic embeddings and measure the word weights within topics by linear transformation between topic and word embeddings, resulting in redundant and inaccurate topic representations. To solve this problem, we propose a novel neural topic model, namely G enerative M odel with N onlinear N eural T opics (GMnnt). The insight of GMnnt is to replace the topic embeddings with neural networks of topics, named neural topic, so as to capture nonlinear relationships between words in the embedding space, enabling to induce more accurate topic representations. We derive the inference process of GMnnt under the framework of neural variational inference. Extensive empirical studies have been conducted on several widely used collections of documents, including datasets of both short texts and normal long texts. The experimental results validate that GMnnt can output more semantically coherent topics compared with traditional topic models and neural topic models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

Notes

  1. We will clarify the definition of variational neural network in Section 3.2.

  2. In this work, we fix S to 1 as suggested in [26].

  3. http://cogcomp.cs.illinois.edu/Data/QA/QC/

  4. https://github.com/jacoxu/STC2

  5. dataset of paper abstracts

  6. http://mulan.sourceforge.net/datasets-mlc.html

  7. https://bitbucket.org/franrruiz/data-nyt-largev-6/src/master/

  8. https://nlp.stanford.edu/projects/glove/

  9. https://github.com/blei-lab/lda-c

  10. https://github.com/ysmiao/nvdm

  11. https://github.com/akashgit/autoencoding_vi_for_topic_models

  12. https://github.com/linkstrife/NVDM-GSM

  13. https://github.com/adjidieng/ETM

  14. https://github.com/dice-group/Palmetto/wiki/Coherences

References

  1. Batmanghelich, K., Saeedi, A., Narasimhan, K., Gershman, S.: Nonparametric spherical topic modeling with word embeddings. In: Annual Meeting of the Association for Computational Linguistics, pp. 537–542 (2016)

  2. Blei, D.M.: Probabilistic topic models. Communications of The ACM 55(4), 77–84 (2012)

    Article  Google Scholar 

  3. Blei, D.M., Kucukelbir, A., Mcauliffe, J.: Variational inference: a review for statisticians. J. Am. Stat. Assoc. 112(518), 859–877 (2017)

    Article  MathSciNet  Google Scholar 

  4. Blei, D.M., Lafferty, J.: Dynamic topic models. In: International Conference on Machine Learning, pp. 113–120 (2006)

  5. Blei, D.M., Lafferty, J.: A correlated topic model of science. The Annals of Applied Statistics 1(1), 17–35 (2007)

    Article  MathSciNet  Google Scholar 

  6. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. J. Mach. Learn. Res. 3(Jan), 993–1022 (2003)

    MATH  Google Scholar 

  7. Boyd-Graber, J., Hu, Y., Mimno, D.: Applications of topic models. Foundations and Trends in Information Retrieval 11(2-3), 143–296 (2017)

    Article  Google Scholar 

  8. Burkhardt, S., Kramer, S.: Decoupling sparsity and smoothness in the Dirichlet variational autoencoder topic model. J. Mach. Learn. Res. 20(131), 1–27 (2019)

    MathSciNet  MATH  Google Scholar 

  9. Cheng, X., Yan, X., Lan, Y., Guo, J.: BTM: topic modeling over short texts. IEEE Trans. Knowl. Data Eng. 26(12), 2928–2941 (2014)

    Article  Google Scholar 

  10. Cong, Y., Chen, B., Liu, H., M.Z.: Deep latent dirichlet allocation with topic-layer-adaptive stochastic gradient riemannian MCMC. In: International Conference on Machine Learning, pp. 864–873 (2017)

  11. Das, R., Zaheer, M., Dyer, C.: Gaussian LDA for topic models with word embeddings. In: International Joint Conference on Natural Language Processing, pp. 795–804 (2015)

  12. Dieng, A.B., Ruiz, F.J.R., Blei, D.M.: The dynamic embedded topic model. arXiv:1907.05545 (2019)

  13. Dieng, A.B., Ruiz, F.J.R., Blei, D.M.: Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics 8, 439–453 (2020)

    Article  Google Scholar 

  14. Dozat, T.: Incorporating nesterov momentum into adam. In: International Conference on Learning Representations Workshop (2016)

  15. Duchi, J.C., Hazan, E., Singer, Y.: Adaptive subgradient methods for online learning and stochastic optimization. J. Mach. Learn. Res. 12(7), 2121–2159 (2011)

    MathSciNet  MATH  Google Scholar 

  16. Feng, J., Rao, Y., Xie, H., Wang, F., Li, Q.: User group based emotion detection and topic discovery over short text. World Wide Web 23, 1553–1587 (2020)

    Article  Google Scholar 

  17. Gershman, S., Goodman, N.D.: Amortized inference in probabilistic reasoning. In: Annual Meeting of the Cognitive Science Society (2014)

  18. Griffiths, T.L., Steyvers, M.: Finding scientific topics. Proc. Natl. Acad. Sci. U.S.A. 101(suppl 1), 5228–5235 (2004)

    Article  Google Scholar 

  19. Gui, L., Leng, J., Pergola, G., Zhou, Y., Xu, R., He, Y.: Neural topic model with reinforcement learning. In: Conference on Empirical Methods in Natural Language Processing, pp. 3478–3483 (2019)

  20. Hoffman, M.D., Bach, F., Blei, D.M.: Online learning for latent dirichlet allocation. In: Neural Information Processing Systems, pp. 856–864 (2010)

  21. Hoffman, M.D., Blei, D.M., Wang, C., Paisley, J.: Stochastic variational inference. J. Mach. Learn. Res. 14(1), 1303–1347 (2013)

    MathSciNet  MATH  Google Scholar 

  22. Isonuma, M., Mori, J., Bollegala, D., Sakata, I.: Tree-structured neural topic model. In: Annual Meeting of the Association for Computational Linguistics, pp. 800–806 (2020)

  23. Jiang, H., Zhou, R., Zhang, L., Wang, H., Zhang, Y.: Sentence level topic models for associated topics extraction. World Wide Web 22(6), 2545–2560 (2019)

    Article  Google Scholar 

  24. Jordan, M.I., Ghahramani, Z., Jaakkola, T.S., Saul, L.K.: An introduction to variational methods for graphical models. Mach. Learn. 37(2), 105–161 (1999)

    Article  Google Scholar 

  25. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations (2015)

  26. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: International Conference on Learning Representations (2014)

  27. Li, W., Mccallum, A.: Pachinko allocation: DAG-structured mixture models of topic correlations. In: International Conference on Machine Learning, pp. 577–584 (2006)

  28. Li, X., Chi, J., Li, C., Ouyang, J., Fu, B.: Integrating topic modeling with word embeddings by mixtures of VMFs. In: International Conference on Computational Linguistics, pp. 151–160 (2016)

  29. Li, X., Ouyang, J., Zhou, X.: Sparse hybrid variational-gibbs algorithm for latent dirichlet allocation. In: SIAM International Conference on Data Mining, pp. 729–737 (2016)

  30. Li, X., Zhang, A., Li, C., Ouyang, J., Cai, Y.: Exploring coherent topics by topic modeling with term weighting. Inform. Process. Manage. 54(6), 1345–1358 (2018)

    Article  Google Scholar 

  31. Li, X., Zhang, J., Ouyang, J.: Dirichlet multinomial mixture with variational manifold regularization: Topic modeling over short texts. In: AAAI Conference on of Artificial Intelligence, pp. 7884–7891 (2019)

  32. Li, Z., Wang, X., Li, J., Zhang, Q.: Deep attributed network representation learning of complex coupling and interaction. Knowl.-Based Syst. 212, 106,618 (2021)

    Article  Google Scholar 

  33. Liu, L., Huang, H., Gao, Y., Zhang, Y., Wei, X.: Neural variational correlated topic modeling. In: The Web Conference, pp. 1142–1152 (2019)

  34. Miao, Y., Grefenstette, E., Blunsom, P.: Discovering discrete latent topics with neural variational inference. In: International Conference on Machine Learning, pp. 2410–2419 (2017)

  35. Miao, Y., Yu, L., Blunsom, P.: Neural variational inference for text processing. In: International Conference on Machine Learning, pp. 1727–1736 (2016)

  36. Mikolov, T., Chen, K., Corrado, G.S., Dean, J.: Efficient estimation of word representations in vector space. In: International Conference on Learning Representations (2013)

  37. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Neural Information Processing Systems, pp. 3111–3119 (2013)

  38. Mimno, D.M., Hoffman, M.D., Blei, D.M.: Sparse stochastic inference for latent dirichlet allocation. In: International Conference on Machine Learning, pp. 1515–1522 (2012)

  39. Mnih, A., Gregor, K.: Neural variational inference and learning in belief networks. In: International Conference on Machine Learning, pp. 1791–1799 (2014)

  40. Naesseth, C.A., Ruiz, F.J.R., Linderman, S.W., Blei, D.M.: Reparameterization gradients through acceptance-rejection sampling algorithms. In: International Conference on Artificial Intelligence and Statistics, pp. 489–498 (2017)

  41. Nan, F., Ding, R., Nallapati, R., Xiang, B.: Topic modeling with wasserstein autoencoders. In: Annual Meeting of the Association for Computational Linguistics, pp. 6345–6381 (2019)

  42. Pennington, J., Socher, R., Manning, C.D.: GloVe: Global vectors for word representation. In: Conference on Empirical Methods in Natural Language Processing, pp. 1532–1543 (2014)

  43. Pergola, G., Gui, L., He, Y.: Tdam: a topic-dependent attention model for sentiment analysis. Inform. Process. Manage. 56(6), 102,084 (2019)

    Article  Google Scholar 

  44. Rashid, J., Shah, S.M.A., Irtaza, A.: Fuzzy topic modeling approach for text mining over short text. Inform. Process. Manage. 56(6), 102,060 (2019)

    Article  Google Scholar 

  45. Rezende, D.J., Mohamed, S., Wierstra, D.: Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In: International Conference on Machine Learning, pp. 1278–1286 (2014)

  46. Röder, M., Both, A., Hinneburg, A.: Exploring the Space of Topic Coherence Measures. In: International Conference on Web Search and Data Mining, pp. 399–408 (2015)

  47. Srivastava, A., Sutton, C.A.: Autoencoding Variational Inference for Topic Models. In: International Conference on Learning Representations (2017)

  48. Teh, Y.W., Jordan, M.I., Beal, M.J., Blei, D.M.: Hierarchical Dirichlet processes. J. Am. Stat. Assoc. 101(476), 1566–1581 (2006)

    Article  MathSciNet  Google Scholar 

  49. Titsias, M., Lázaro-Gredilla, M.: Doubly stochastic variational bayes for non-conjugate inference. In: International Conference on Machine Learning, pp. 4056–4069 (2014)

  50. Tolstikhin, I., Bousquet, O., Gelly, S., Schoelkopf, B.: Wasserstein auto-encoders. In: International Conference on Learning Representations (2018)

  51. Wang, C., Blei, D.M., Heckerman, D.: Continuous time dynamic topic model. In: Uncertainty in Artificial Intelligence, pp. 579–586 (2008)

  52. Wang, R., Zhou, D., He, Y.: Atm: Adversarial-neural topic model. Inform. Process. Manage. 56(6), 102,098 (2019)

    Article  Google Scholar 

  53. Wang, Y., Li, X., Ouyang, J.: Layer-assisted neural topic modeling over document networks. In: International Joint Conference on Artificial Intelligence, pp. 3148–3154 (2021)

  54. Zhang, H., Chen, B., Guo, D., Zhou, M.: WHAI: Weibull hybrid autoencoding inference for deep topic modeling. In: International Conference on Learning Representations (2018)

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (NSFC) [No.61876071] and Scientific and Technological Developing Scheme of Jilin Province [No.20180201003SF, No.20190701031GH] and Energy Administration of Jilin Province [No.3D516L921421].

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Jilin University, Jilin, China

    Yiming Wang, Ximing Li, Jihong Ouyang, Zeqi Guo & Yimeng Wang

  2. Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Jilin, China

    Yiming Wang, Ximing Li, Jihong Ouyang, Zeqi Guo & Yimeng Wang

Authors
  1. Yiming Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Ximing Li
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Jihong Ouyang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Zeqi Guo
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Yimeng Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Ximing Li.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Y., Li, X., Ouyang, J. et al. Extracting nonlinear neural topics with neural variational bayes. World Wide Web 25, 131–149 (2022). https://doi.org/10.1007/s11280-021-00970-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11280-021-00970-8

Keywords