skip to main content
10.1145/3308558.3313561acmotherconferencesArticle/Chapter ViewAbstractPublication PagesthewebconfConference Proceedingsconference-collections
research-article

Neural Variational Correlated Topic Modeling

Published: 13 May 2019 Publication History

Abstract

With the rapid development of the Internet, millions of documents, such as news and web pages, are generated everyday. Mining the topics and knowledge on them has attracted a lot of interest on both academic and industrial areas. As one of the prevalent unsupervised data mining tools, topic models are usually explored as probabilistic generative models for large collections of texts. Traditional probabilistic topic models tend to find a closed form solution of model parameters and approach the intractable posteriors via approximation methods, which usually lead to the inaccurate inference of parameters and low efficiency when it comes to a quite large volume of data. Recently, an emerging trend of neural variational inference can overcome the above issues, which offers a scalable and powerful deep generative framework for modeling latent topics via neural networks. Interestingly, a common assumption for the most neural variational topic models is that topics are independent and irrelevant to each other. However, this assumption is unreasonable in many practical scenarios. In this paper, we propose a novel Centralized Transformation Flow to capture the correlations among topics by reshaping topic distributions. Furthermore, we present the Transformation Flow Lower Bound to improve the performance of the proposed model. Extensive experiments on two standard benchmark datasets have well-validated the effectiveness of the proposed approach.

References

[1]
Amrudin Agovic and Arindam Banerjee. 2012. Gaussian Process Topic Models. CoRRabs/1203.3462(2012). http://arxiv.org/abs/1203.3462
[2]
Loulwah AlSumait, Daniel Barbará, and Carlotta Domeniconi. 2008. On-line lda: Adaptive topic models for mining text streams with applications to topic detection and tracking. In Data Mining, 2008. ICDM'08. Eighth IEEE International Conference on. IEEE, 3-12.
[3]
Christophe Andrieu, Nando De Freitas, Arnaud Doucet, and Michael I Jordan. 2003. An introduction to MCMC for machine learning. Machine learning50, 1-2 (2003), 5-43.
[4]
James Bergstra and Yoshua Bengio. 2012. Random search for hyper-parameter optimization. Journal of Machine Learning Research13, Feb (2012), 281-305.
[5]
Christian H Bischof and Xiaobai Sun. 1994. On orthogonal block elimination. Preprint MCS-P450-0794, Mathematics and Computer Science Division, Argonne National Laboratory(1994).
[6]
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. 2017. Variational inference: A review for statisticians. J. Amer. Statist. Assoc.112, 518 (2017), 859-877.
[7]
David M. Blei and John D. Lafferty. 2005. Correlated Topic Models. In Advances in Neural Information Processing Systems 18 {Neural Information Processing Systems, NIPS 2005, December 5-8, 2005, Vancouver, British Columbia, Canada}. 147-154. http://papers.nips.cc/paper/2906-correlated-topic-models
[8]
David M. Blei and John D. Lafferty. 2005. Correlated Topic Models. (2005), 147-154. http://papers.nips.cc/paper/2906-correlated-topic-models
[9]
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2001. Latent Dirichlet Allocation. In Advances in Neural Information Processing Systems 14 {Neural Information Processing Systems: Natural and Synthetic, NIPS 2001, December 3-8, 2001, Vancouver, British Columbia, Canada}, Thomas G. Dietterich, Suzanna Becker, and Zoubin Ghahramani (Eds.). MIT Press, 601-608. http://papers.nips.cc/paper/2070-latent-dirichlet-allocation
[10]
Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST)2, 3(2011), 27.
[11]
Rajarshi Das, Manzil Zaheer, and Chris Dyer. 2015. Gaussian lda for topic models with word embeddings. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Vol. 1. 795-804.
[12]
Philipp Hennig, David Stern, Ralf Herbrich, and Thore Graepel. 2012. Kernel topic models. In Artificial Intelligence and Statistics. 511-519.
[13]
Geoffrey E Hinton, Peter Dayan, Brendan J Frey, and Radford M Neal. 1995. The” wake-sleep” algorithm for unsupervised neural networks. Science268, 5214 (1995), 1158-1161.
[14]
Matthew Hoffman, Francis R Bach, and David M Blei. 2010. Online learning for latent dirichlet allocation. In advances in neural information processing systems. 856-864.
[15]
Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley. 2013. Stochastic variational inference. The Journal of Machine Learning Research14, 1 (2013), 1303-1347.
[16]
Liangjie Hong and Brian D Davison. 2010. Empirical study of topic modeling in twitter. In Proceedings of the first workshop on social media analytics. ACM, 80-88.
[17]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980(2014).
[18]
Diederik P. Kingma, Tim Salimans, and Max Welling. 2016. Improving Variational Inference with Inverse Autoregressive Flow. CoRRabs/1606.04934(2016).
[19]
Diederik P. Kingma and Max Welling. 2013. Auto-Encoding Variational Bayes. CoRRabs/1312.6114(2013). http://arxiv.org/abs/1312.6114
[20]
Jey Han Lau, David Newman, and Timothy Baldwin. 2014. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics. 530-539.
[21]
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research9, Nov (2008), 2579-2605.
[22]
Yishu Miao, Edward Grefenstette, and Phil Blunsom. 2017. Discovering Discrete Latent Topics with Neural Variational Inference. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017(Proceedings of Machine Learning Research), Doina Precup and Yee Whye Teh (Eds.). Vol. 70. PMLR, 2410-2419. http://proceedings.mlr.press/v70/miao17a.html
[23]
Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neural Variational Inference for Text Processing. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016(JMLR Workshop and Conference Proceedings), Maria-Florina Balcan and Kilian Q. Weinberger (Eds.). Vol. 48. JMLR.org, 1727-1736. http://jmlr.org/proceedings/papers/v48/miao16.html
[24]
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems. 3111-3119.
[25]
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 1532-1543.
[26]
Kaare Brandt Petersen, Michael Syskind Pedersen, 2008. The matrix cookbook. Technical University of Denmark7, 15 (2008), 510.
[27]
Ian Porteous, David Newman, Alexander Ihler, Arthur Asuncion, Padhraic Smyth, and Max Welling. 2008. Fast collapsed gibbs sampling for latent dirichlet allocation. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 569-577.
[28]
Daniel Ramage, David Hall, Ramesh Nallapati, and Christopher D Manning. 2009. Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1. Association for Computational Linguistics, 248-256.
[29]
Radim Rehurek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. ELRA, Valletta, Malta, 45-50. http://is.muni.cz/publication/884893/en.
[30]
Danilo Jimenez Rezende and Shakir Mohamed. 2015. Variational Inference with Normalizing Flows. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015(JMLR Workshop and Conference Proceedings), Francis R. Bach and David M. Blei (Eds.). Vol. 37. JMLR.org, 1530-1538. http://jmlr.org/proceedings/papers/v37/rezende15.html
[31]
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014(JMLR Workshop and Conference Proceedings), Vol. 32. JMLR.org, 1278-1286. http://jmlr.org/proceedings/papers/v32/rezende14.html
[32]
Tim Salimans, Diederik Kingma, and Max Welling. 2015. Markov chain monte carlo and variational inference: Bridging the gap. In International Conference on Machine Learning. 1218-1226.
[33]
Tian Shi, Kyeongpil Kang, Jaegul Choo, and Chandan K Reddy. 2018. Short-Text Topic Modeling via Non-negative Matrix Factorization Enriched with Local Word-Context Correlations. In Proceedings of the 2018 World Wide Web Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 1105-1114.
[34]
Akash Srivastava and Charles Sutton. 2017. Autoencoding Variational Inference For Topic Models. arXiv preprint arXiv:1703.01488(2017).
[35]
Mark Steyvers, Padhraic Smyth, Michal Rosen-Zvi, and Thomas Griffiths. 2004. Probabilistic author-topic models for information discovery. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 306-315.
[36]
Xiaobai Sun and Christian Bischof. 1995. A basis-kernel representation of orthogonal matrices. SIAM journal on matrix analysis and applications16, 4(1995), 1184-1196.
[37]
Jakub M. Tomczak and Max Welling. 2016. Improving Variational Auto-Encoders using Householder Flow. CoRRabs/1611.09630(2016). http://arxiv.org/abs/1611.09630
[38]
Guangxu Xun, Yaliang Li, Wayne Xin Zhao, Jing Gao, and Aidong Zhang. 2017. A Correlated Topic Model Using Word Embeddings. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017. 4207-4213.
[39]
Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto. 2007. On early stopping in gradient descent learning. Constructive Approximation26, 2 (2007), 289-315.

Cited By

View all
  • (2025)Bridging insight gaps in topic dependency discovery with a knowledge-inspired topic modelInformation Processing & Management10.1016/j.ipm.2024.10391162:1(103911)Online publication date: Jan-2025
  • (2024)The Generative Generic-Field Design Method Based on Design Cognition and Knowledge ReasoningSustainability10.3390/su1622984116:22(9841)Online publication date: 12-Nov-2024
  • (2024)Strategic Integration of Context for Fine-Tuning Topic Model Performance2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC61105.2024.00058(366-375)Online publication date: 2-Jul-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
WWW '19: The World Wide Web Conference
May 2019
3620 pages
ISBN:9781450366748
DOI:10.1145/3308558
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

  • IW3C2: International World Wide Web Conference Committee

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 May 2019

Permissions

Request permissions for this article.

Check for updates

Author Tag

  1. Natural language processing;topic model;neural variational inference

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Conference

WWW '19
WWW '19: The Web Conference
May 13 - 17, 2019
CA, San Francisco, USA

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)40
  • Downloads (Last 6 weeks)4
Reflects downloads up to 06 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Bridging insight gaps in topic dependency discovery with a knowledge-inspired topic modelInformation Processing & Management10.1016/j.ipm.2024.10391162:1(103911)Online publication date: Jan-2025
  • (2024)The Generative Generic-Field Design Method Based on Design Cognition and Knowledge ReasoningSustainability10.3390/su1622984116:22(9841)Online publication date: 12-Nov-2024
  • (2024)Strategic Integration of Context for Fine-Tuning Topic Model Performance2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC)10.1109/COMPSAC61105.2024.00058(366-375)Online publication date: 2-Jul-2024
  • (2024)Dependency-Aware Neural Topic ModelInformation Processing & Management10.1016/j.ipm.2023.10353061:1(103530)Online publication date: Jan-2024
  • (2024)A survey on neural topic models: methods, applications, and challengesArtificial Intelligence Review10.1007/s10462-023-10661-757:2Online publication date: 25-Jan-2024
  • (2024)Big topic modeling based on a two-level hierarchical latent Beta-Liouville allocation for large-scale data and parameter streamingPattern Analysis and Applications10.1007/s10044-024-01213-y27:1Online publication date: 28-Feb-2024
  • (2024)CSGTM: Capsule Semantic Graph-Guided Latent Community Topics DiscoveryWeb and Big Data10.1007/978-981-97-7238-4_19(292-307)Online publication date: 28-Aug-2024
  • (2023)Neural Personalized Topic Modeling for Mining User Preferences on Social MediaProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614987(1545-1555)Online publication date: 21-Oct-2023
  • (2023)Neural Variational Gaussian Mixture Topic ModelACM Transactions on Asian and Low-Resource Language Information Processing10.1145/357858322:4(1-18)Online publication date: 25-Mar-2023
  • (2023)A Topic-aware Summarization Framework with Different Modal Side InformationProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591630(1416-1425)Online publication date: 19-Jul-2023
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media