Skip to main content
SpringerLink
Log in

Encouraging Sparsity in Neural Topic Modeling with Non-Mean-Field Inference

  • Conference paper
  • First Online:
Machine Learning and Knowledge Discovery in Databases: Research Track (ECML PKDD 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14172))

Abstract

Topic modeling is a popular method for discovering semantic information from textual data, with latent Dirichlet allocation (LDA) being a representative model. Recently, researchers have explored the use of variational autoencoders (VAE) to improve the performance of LDA. However, there remain two major limitations: (1) the Dirichlet prior is inadequate to extract precise semantic information in VAE-LDA models, as it introduces a trade-off between the topic quality and the sparsity of representations; (2) new variants of VAE-LDA models with auxiliary variables generally ignore the correlation between latent variables in the inference process due to the Mean-Field assumption. To address these issues, in this paper, we propose a Sparsity Reinforced and Non-Mean-Field Topic Model (SpareNTM) with a bank of auxiliary Bernoulli variables in the generative process of LDA to further model the sparsity of document representations. Thus individual documents are forced to focus on a subset of topics by a corresponding Bernoulli topic selector. Then, instead of applying the mean-field assumption for the posterior approximation, we take full advantage of VAE to realize a non-mean-field approximation, which succeeds in preserving the connection of latent variables. Experiment results on three datasets (20NewsGroup, Wikitext-103, and SearchSnippets) show that our model outperforms recent topic models in terms of both topic quality and sparsity.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The code is available at https://github.com/Nazzcjy/SpareNTM.

  2. 2.

    https://github.com/Nazzcjy/SpareNTM.

  3. 3.

    The Gumbel(0, 1) distribution can be sampled using inverse transform sampling by drawing \(u\sim \text{ Uniform(0, } \text{1) }\) and computing \(g=-\log (-\log (u))\).

  4. 4.

    https://github.com/nguyentthong/CLNTM.

  5. 5.

    https://github.com/qiang2100/STTM.

  6. 6.

    https://github.com/akashgit/autoencoding_vi_for_topic_models.

  7. 7.

    https://github.com/sophieburkhardt/dirichlet-vae-topic-models.

  8. 8.

    https://github.com/dallascard/SCHOLAR.

  9. 9.

    https://github.com/bobxwu/NQTM.

  10. 10.

    Instead of using a sliding window, we consider a whole document to identify co-occurrence.

  11. 11.

    http://deepdive.stanford.edu/opendata/.

References

  1. Blei, D.M., Kucukelbir, A., McAuliffe, J.D.: Variational inference: a review for statisticians. J. Am. Stat. Assoc. 112(518), 859–877 (2017)

    Article  MathSciNet  Google Scholar 

  2. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent Dirichlet allocation. JMLR 3(Jan), 993–1022 (2003)

    Google Scholar 

  3. Burkhardt, S., Kramer, S.: Decoupling sparsity and smoothness in the Dirichlet variational autoencoder topic model. JMLR 20, 131:1–131:27 (2019)

    Google Scholar 

  4. Card, D., Tan, C., Smith, N.A.: Neural models for documents with metadata. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, 15–20 July 2018, pp. 2031–2040. Association for Computational Linguistics (2018)

    Google Scholar 

  5. Dieng, A.B., Wang, C., Gao, J., Paisley, J.W.: TopiCRNN: a recurrent neural network with long-range semantic dependency. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, 24–26 April 2017, Conference Track Proceedings. OpenReview.net (2017)

    Google Scholar 

  6. Dieng, A.B., Ruiz, F.J.R., Blei, D.M.: Topic modeling in embedding spaces. Trans. Assoc. Comput. Linguist. 8, 439–453 (2020)

    Article  Google Scholar 

  7. Drefs, J., Guiraud, E., Lücke, J.: Evolutionary variational optimization of generative models. J. Mach. Learn. Res. 23, 21:1–21:51 (2022)

    Google Scholar 

  8. Drefs, J., Guiraud, E., Panagiotou, F., Lücke, J.: Direct evolutionary optimization of variational autoencoders with binary latents. In: Amini, M.R., Canu, S., Fischer, A., Guns, T., Kralj Novak, P., Tsoumakas, G. (eds.) ECML PKDD 2022. LNCS, vol. 13715, pp. 357–372. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-26409-2_22

    Chapter  Google Scholar 

  9. Fallah, K., Rozell, C.J.: Variational sparse coding with learned thresholding. In: ICML. Proceedings of Machine Learning Research, vol. 162, pp. 6034–6058. PMLR (2022)

    Google Scholar 

  10. Feng, J., Zhang, Z., Ding, C., Rao, Y., Xie, H., Wang, F.L.: Context reinforced neural topic modeling over short texts. Inf. Sci. 607, 79–91 (2022)

    Article  Google Scholar 

  11. Griffiths, T.L., Steyvers, M.: Finding scientific topics. Proc. Natl. Acad. Sci. 101(suppl_1), 5228–5235 (2004)

    Google Scholar 

  12. Jang, E., Gu, S., Poole, B.: Categorical reparameterization with gumbel-softmax. In: ICLR (2017)

    Google Scholar 

  13. Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: ICLR (2014)

    Google Scholar 

  14. Lang, K.: Newsweeder: learning to filter netnews. In: Machine Learning, Proceedings of the Twelfth International Conference on Machine Learning, Tahoe City, California, USA, 9–12 July 1995, pp. 331–339 (1995). https://doi.org/10.1016/b978-1-55860-377-6.50048-7

  15. Lau, J.H., Newman, D., Baldwin, T.: Machine reading tea leaves: automatically evaluating topic coherence and topic model quality. In: EACL, pp. 530–539 (2014)

    Google Scholar 

  16. Lin, T., Hu, Z., Guo, X.: Sparsemax and relaxed Wasserstein for topic sparsity. In: WSDM, pp. 141–149 (2019)

    Google Scholar 

  17. Merity, S., Xiong, C., Bradbury, J., Socher, R.: Pointer sentinel mixture models. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, 24–26 April 2017, Conference Track Proceedings (2017)

    Google Scholar 

  18. Miao, Y., Grefenstette, E., Blunsom, P.: Discovering discrete latent topics with neural variational inference. In: ICML, vol. 70, pp. 2410–2419 (2017)

    Google Scholar 

  19. Miao, Y., Yu, L., Blunsom, P.: Neural variational inference for text processing. In: ICML, vol. 48, pp. 1727–1736 (2016)

    Google Scholar 

  20. Mitchell, T.J., Beauchamp, J.J.: Bayesian variable selection in linear regression. J. Am. Stat. Assoc. 83(404), 1023–1032 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  21. Naesseth, C.A., Ruiz, F.J.R., Linderman, S.W., Blei, D.M.: Reparameterization gradients through acceptance-rejection sampling algorithms. In: AISTATS. Proceedings of Machine Learning Research, vol. 54, pp. 489–498 (2017)

    Google Scholar 

  22. Nan, F., Ding, R., Nallapati, R., Xiang, B.: Topic modeling with Wasserstein autoencoders. In: Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, 28 July–2 August 2019, pp. 6345–6381 (2019)

    Google Scholar 

  23. Nguyen, T., Luu, A.T.: Contrastive learning for neural topic model. In: Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, 6–14 December 2021, Virtual, pp. 11974–11986 (2021)

    Google Scholar 

  24. Ning, X., et al.: Nonparametric topic modeling with neural inference. Neurocomputing 399, 296–306 (2020)

    Article  Google Scholar 

  25. Peng, M., et al.: Neural sparse topical coding. In: ACL, pp. 2332–2340 (2018)

    Google Scholar 

  26. Phan, X.H., Nguyen, L., Horiguchi, S.: Learning to classify short and sparse text & web with hidden topics from large-scale data collections. In: Proceedings of the 17th International Conference on World Wide Web, WWW 2008, Beijing, China, 21–25 April 2008, pp. 91–100 (2008). https://doi.org/10.1145/1367497.1367510

  27. Rezaee, M., Ferraro, F.: A discrete variational recurrent topic model without the reparametrization trick. In: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (eds.) Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, 6–12 December 2020, Virtual (2020)

    Google Scholar 

  28. Song, Z., Hu, Y., Verma, A., Buckeridge, D.L., Li, Y.: Automatic phenotyping by a seed-guided topic model. In: KDD, pp. 4713–4723. ACM (2022)

    Google Scholar 

  29. Srivastava, A., Sutton, C.: Autoencoding variational inference for topic models. In: ICLR (2017)

    Google Scholar 

  30. Srivastava, A., Sutton, C.: Variational inference in pachinko allocation machines. CoRR (2018)

    Google Scholar 

  31. Tian, R., Mao, Y., Zhang, R.: Learning VAE-LDA models with rounded reparameterization trick. In: EMNLP, pp. 1315–1325 (2020)

    Google Scholar 

  32. Tonolini, F., Jensen, B.S., Murray-Smith, R.: Variational sparse coding. In: UAI. Proceedings of Machine Learning Research, vol. 115, pp. 690–700. AUAI Press (2019)

    Google Scholar 

  33. Turner, R.E., Sahani, M.: Two problems with variational expectation maximisation for time-series models. In: Barber, D., Cemgil, T., Chiappa, S. (eds.) Bayesian Time Series Models, chap. 5, pp. 109–130. Cambridge University Press (2011)

    Google Scholar 

  34. Wang, C., Blei, D.M.: Decoupling sparsity and smoothness in the discrete hierarchical Dirichlet process. In: NeurIPS, pp. 1982–1989 (2009)

    Google Scholar 

  35. Wang, D., et al.: Representing mixtures of word embeddings with mixtures of topic embeddings. In: ICLR. OpenReview.net (2022)

    Google Scholar 

  36. Wang, R., et al.: Neural topic modeling with bidirectional adversarial training. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, 5–10 July 2020, pp. 340–350 (2020)

    Google Scholar 

  37. Wang, R., Zhou, D., He, Y.: ATM: adversarial-neural topic model. Inf. Process. Manag. 56(6) (2019)

    Google Scholar 

  38. Williamson, S., Wang, C., Heller, K.A., Blei, D.M.: The IBP compound Dirichlet process and its application to focused topic modeling. In: ICML, pp. 1151–1158 (2010)

    Google Scholar 

  39. Wu, X., Li, C., Zhu, Y., Miao, Y.: Short text topic modeling with topic distribution quantization and negative sampling decoder. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP, pp. 1772–1782 (2020)

    Google Scholar 

  40. Xu, J., Xu, B., Wang, P., Zheng, S., Tian, G., Zhao, J.: Self-taught convolutional neural networks for short text clustering. Neural Netw. 88, 22–31 (2017)

    Article  Google Scholar 

  41. Zhao, H., Phung, D.Q., Huynh, V., Jin, Y., Du, L., Buntine, W.L.: Topic modelling meets deep neural networks: a survey. In: IJCAI, pp. 4713–4720 (2021)

    Google Scholar 

  42. Zhu, J., Xing, E.P.: Sparse topical coding. In: UAI, pp. 831–838 (2011)

    Google Scholar 

Download references

Acknowledgment

This work was supported by the National Natural Science Foundation of China (61972261), Natural Science Foundation of Guangdong Province (2023A1515011667), Key Basic Research Foundation of Shenzhen (JCYJ20220818100205012), and Basic Research Foundations of Shenzhen (JCYJ20210324093609026).

Author information

Authors and Affiliations

  1. College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, China

    Jiayao Chen, Rui Wang, Jueying He & Mark Junjie Li

Authors
  1. Jiayao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rui Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jueying He
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mark Junjie Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mark Junjie Li .

Editor information

Editors and Affiliations

  1. University of Michigan, Ann Arbor, MI, USA

    Danai Koutra

  2. University of Vienna, Vienna, Austria

    Claudia Plant

  3. Max Planck Institute for Software Systems, Kaiserslautern, Germany

    Manuel Gomez Rodriguez

  4. Politecnico di Torino, Turin, Italy

    Elena Baralis

  5. CENTAI, Turin, Italy

    Francesco Bonchi

Ethics declarations

Ethical Statement

This paper doesn’t involve unethical tasks. All the experiment was done in public datasets.

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chen, J., Wang, R., He, J., Li, M.J. (2023). Encouraging Sparsity in Neural Topic Modeling with Non-Mean-Field Inference. In: Koutra, D., Plant, C., Gomez Rodriguez, M., Baralis, E., Bonchi, F. (eds) Machine Learning and Knowledge Discovery in Databases: Research Track. ECML PKDD 2023. Lecture Notes in Computer Science(), vol 14172. Springer, Cham. https://doi.org/10.1007/978-3-031-43421-1_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-43421-1_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-43420-4

  • Online ISBN: 978-3-031-43421-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation