Beyond Labels and Topics: Discovering Causal Relationships in Neural Topic Modeling
Pages 4460 - 4469
Abstract
Topic models that can take advantage of labels are broadly used in identifying interpretable topics from textual data. However, existing topic models tend to merely view labels as names of topic clusters or as categories of texts, thereby neglecting the potential causal relationships between supervised information and latent topics, as well as within these elements themselves. In this paper, we focus on uncovering possible causal relationships both between and within the supervised information and latent topics to better understand the mechanisms behind the emergence of the topics and the labels. To this end, we propose Causal Relationship-Aware Neural Topic Model (CRNTM), a novel neural topic model that can automatically uncover interpretable causal relationships between and within supervised information and latent topics, while concurrently discovering high-quality topics. In CRNTM, both supervised information and latent topics are treated as nodes, with the causal relationships represented as directed edges in a Directed Acyclic Graph (DAG). A Structural Causal Model (SCM) is employed to model the DAG. Experiments are conducted on three public corpora with different types of labels. Experimental results show that the discovered causal relationships are both reliable and interpretable, and the learned topics are of high quality comparing with eight start-of-the-art topic model baselines.
Supplemental Material
MP4 File
Supplemental video
- Download
- 31.17 MB
References
[1]
Pritom Saha Akash, Jie Huang, and Kevin Chen-Chuan Chang. 2022. Coordinated Topic Modeling. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 9831--9843.
[2]
Ruina Bai, Ruizhang Huang, Yongbin Qin, Yanping Chen, and Chuan Lin. 2023. HVAE: A deep generative model via hierarchical variational auto-encoder for multi-view document modeling. Information Sciences, Vol. 623 (2023), 40--55.
[3]
Federico Bianchi, Silvia Terragni, and Dirk Hovy. 2021. Pre-training is a Hot Topic: Contextualized Document Embeddings Improve Topic Coherence. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). Association for Computational Linguistics, Online, 759--766.
[4]
Christopher M Bishop. [n.,d.]. Pattern recognition and machine learning. Vol. 4. Springer.
[5]
David M Blei and John D Lafferty. 2005. Correlated topic models. In Proceedings of the 18th International Conference on Neural Information Processing Systems. 147--154.
[6]
David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, Vol. 3 (2003), 993--1022.
[7]
Sophie Burkhardt and Stefan Kramer. 2019. Decoupling Sparsity and Smoothness in the Dirichlet Variational Autoencoder Topic Model. J. Mach. Learn. Res., Vol. 20, 131 (2019), 1--27.
[8]
Dallas Card, Chenhao Tan, and Noah A Smith. 2018. Neural Models for Documents with Metadata. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2031--2040.
[9]
HeGang Chen, Pengbo Mao, Yuyin Lu, and Yanghui Rao. 2023. Nonlinear Structural Equation Model Guided Gaussian Mixture Hierarchical Topic Modeling. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Toronto, Canada, 10377--10390. https://doi.org/10.18653/v1/2023.acl-long.578
[10]
Adji B. Dieng, Francisco J. R. Ruiz, and David M. Blei. 2020. Topic Modeling in Embedding Spaces. Transactions of the Association for Computational Linguistics, Vol. 8 (07 2020), 439--453.
[11]
Anna Glazkova, Yury Egorov, and Maksim Glazkov. 2021. A Comparative Study of Feature Types for Age-Based Text Classification. In Analysis of Images, Social Networks and Texts. Springer International Publishing, Cham, 120--134.
[12]
Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv preprint arXiv:2203.05794 (2022).
[13]
Shoaib Jameel and Wai Lam. 2013. An unsupervised topic segmentation model incorporating word order. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. 203--212.
[14]
Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014.
[15]
Yan Leng, Jian Zhuang, Jie Pan, and Chengli Sun. 2023. Multitask learning for acoustic scene classification with topic-based soft labels and a mutual attention mechanism. Knowledge-Based Systems, Vol. 268 (2023), 110460.
[16]
Dairui Liu, Derek Greene, and Ruihai Dong. 2022b. A Novel Perspective to Look At Attention: Bi-level Attention-based Explainable Topic Modeling for News Classification. In Findings of the Association for Computational Linguistics: ACL 2022. 2280--2290.
[17]
Ziwen Liu, Josep Grau-Bove, and Scott Allan Allan Orr. 2022a. BERT-Flow-VAE: A Weakly-supervised Model for Multi-Label Text Classification. In Proceedings of the 29th International Conference on Computational Linguistics. 1203--1220.
[18]
Haiyi Mao, Hongfu Liu, Jason Xiaotian Dou, and Panayiotis V Benos. 2022. Towards Cross-Modal Causal Structure and Representation Learning. In Machine Learning for Health. PMLR, 120--140.
[19]
Yu Meng, Jiaxin Huang, Guangyuan Wang, Zihan Wang, Chao Zhang, Yu Zhang, and Jiawei Han. 2020a. Discriminative topic mining via category-name guided text embedding. In Proceedings of The Web Conference 2020. 2121--2132.
[20]
Yu Meng, Yunyi Zhang, Jiaxin Huang, Yu Zhang, Chao Zhang, and Jiawei Han. 2020b. Hierarchical topic mining via joint spherical tree and text embedding. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. 1908--1917.
[21]
Yishu Miao, Edward Grefenstette, and Phil Blunsom. 2017. Discovering discrete latent topics with neural variational inference. In Proceedings of the 34th International Conference on Machine Learning-Volume 70. 2410--2419.
[22]
Christian A Naesseth, Francisco JR Ruiz, Scott W Linderman, and David M Blei. 2016. Rejection Sampling Variational Inference. stat, Vol. 1050 (2016), 18.
[23]
Feng Nan, Ran Ding, Ramesh Nallapati, and Bing Xiang. 2019. Topic Modeling with Wasserstein Autoencoders. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 6345--6381.
[24]
Ignavier Ng, Zhuangyan Fang, Shengyu Zhu, Zhitang Chen, and Jun Wang. 2019. Masked gradient-based causal structure learning. arXiv preprint arXiv:1910.08527 (2019).
[25]
Madhur Panwar, Shashank Shailabh, Milan Aggarwal, and Balaji Krishnamurthy. 2021. TAN-N™: Topic Attention Networks for Neural Topic Modeling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 3865--3880.
[26]
Judea Pearl. 2010. Causal inference. Causality: objectives and assessment (2010), 39--58.
[27]
Adler Perotte, Nicholas Bartlett, Noémie Elhadad, and Frank Wood. 2011. Hierarchically supervised latent Dirichlet allocation. In Proceedings of the 24th International Conference on Neural Information Processing Systems. 2609--2617.
[28]
Valerio Perrone, Paul A Jenkins, Dario Spanò, and Yee Whye Teh. 2017. Poisson Random Fields for Dynamic Feature Models. Journal of Machine Learning Research, Vol. 18, 127 (2017), 1--45.
[29]
Dang Pham and Tuan M. V. Le. 2021. Neural Topic Models for Hierarchical Topic Detection and Visualization. In Machine Learning and Knowledge Discovery in Databases. Research Track, Nuria Oliver, Fernando Pérez-Cruz, Stefan Kramer, Jesse Read, and Jose A. Lozano (Eds.). Cham, 35--51.
[30]
Daniel Ramage, David Hall, Ramesh Nallapati, and Christopher D Manning. 2009. Labeled LDA: A supervised topic model for credit attribution in multi-labeled corpora. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1. Association for Computational Linguistics, 248--256.
[31]
Daniel Ramage, Christopher D Manning, and Susan Dumais. 2011. Partially labeled topic models for interpretable text mining. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 457--465.
[32]
Peter Spirtes. 2010. Introduction to causal inference. Journal of Machine Learning Research, Vol. 11, 5 (2010).
[33]
Dhanya Sridhar, III Daumé, Hal, and David Blei. 2022. Heterogeneous Supervised Topic Models. Transactions of the Association for Computational Linguistics, Vol. 10 (06 2022), 732--745.
[34]
Akash Srivastava and Charles A. Sutton. 2017. Autoencoding Variational Inference For Topic Models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24--26, 2017, Conference Track Proceedings.
[35]
Ridam Srivastava, Prabhav Singh, KPS Rana, and Vineet Kumar. 2022. A topic modeled unsupervised approach to single document extractive text summarization. Knowledge-Based Systems, Vol. 246 (2022), 108636.
[36]
Hongda Sun, Quan Tu, Jinpeng Li, and Rui Yan. 2023. ConvNTM: conversational neural topic model. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 13609--13617.
[37]
I Tolstikhin, O Bousquet, S Gelly, and B Schölkopf. 2018. Wasserstein Auto-Encoders. In 6th International Conference on Learning Representations (ICLR 2018). OpenReview. net.
[38]
Federico Tomasi, Praveen Chandar, Gal Levy-Fix, Mounia Lalmas-Roelleke, and Zhenwen Dai. 2020. Stochastic Variational Inference for Dynamic Correlated Topic Models. In Conference on Uncertainty in Artificial Intelligence. PMLR, 859--868.
[39]
Manju Venugopalan and Deepa Gupta. 2022. An enhanced guided LDA model augmented with BERT based semantic strength for aspect term extraction in sentiment analysis. Knowledge-based systems, Vol. 246 (2022), 108668.
[40]
Wei Wang, Bing Guo, Yan Shen, Han Yang, Yaosen Chen, and Xinhua Suo. 2021. Neural labeled LDA: a topic model for semi-supervised document classification. Soft Computing, Vol. 25, 23 (2021), 14561--14571.
[41]
Xinyi Wang and Yi Yang. 2020. Neural topic model with attention for supervised learning. In International Conference on Artificial Intelligence and Statistics. PMLR, 1147--1156.
[42]
Guangxu Xun, Yaliang Li, Wayne Xin Zhao, Jing Gao, and Aidong Zhang. 2017. A correlated topic model using word embeddings. In IJCAI, Vol. 17. 4207--4213.
[43]
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. 2021. CausalVAE: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9593--9602.
[44]
Yi Yang, Kunpeng Zhang, and Yangyang Fan. 2022. sDTM: A Supervised Bayesian Deep Topic Model for Text Analytics. Information Systems Research (2022).
[45]
Yue Yu, Jie Chen, Tian Gao, and Mo Yu. 2019. DAG-GNN: DAG structure learning with graph neural networks. In International Conference on Machine Learning. PMLR, 7154--7163.
[46]
Yu Zhang, Xiusi Chen, Yu Meng, and Jiawei Han. 2021. Hierarchical metadata-aware document categorization under weak supervision. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining. 770--778.
[47]
Xun Zheng, Bryon Aragam, Pradeep Ravikumar, and Eric P Xing. 2018. DAGs with NO TEARS: continuous optimization for structure learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems. 9492--9503.
[48]
Bingshan Zhu, Yi Cai, and Haopeng Ren. 2023. Graph neural topic model with commonsense knowledge. Information Processing & Management, Vol. 60, 2 (2023), 103215.
[49]
Jun Zhu, Amr Ahmed, and Eric P Xing. 2009. MedLDA: maximum margin supervised topic models for regression and classification. In Proceedings of the 26th annual international conference on machine learning. 1257--1264. io
Index Terms
- Beyond Labels and Topics: Discovering Causal Relationships in Neural Topic Modeling
Recommendations
Cycling topic graph learning for neural topic modeling
AbstractTopic models aim to discover a set of latent topics in a textual corpus. Graph Neural Networks (GNNs) have been recently utilized in Neural Topic Models (NTMs) due to their strong capacity to model document representations with the text graph. ...
Reward-Modulated Adversarial Topic Modeling
Database Systems for Advanced ApplicationsAbstractNeural topic models have attracted much attention for their high efficiencies in training, in which, the methods based on variational auto-encoder capture approximative distributions of data, and those based on Generative Adversarial Net (GAN) are ...
Topic modeling methods for short texts: A survey
In the present day, online users are incentivized to engage in short text-based communication. These short texts harbor a significant amount of implicit information, including opinions, topics, and emotions, which are of notable value for both ...
Comments
Information & Contributors
Information
Published In
May 2024
4826 pages
ISBN:9798400701719
DOI:10.1145/3589334
- General Chairs:
- Tat-Seng Chua,
- Chong-Wah Ngo,
- Proceedings Chair:
- Roy Ka-Wei Lee,
- Program Chairs:
- Ravi Kumar,
- Hady W. Lauw
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 May 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- MIIT Program
- the National Natural Science Foundation of China
Conference
WWW '24
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,899 of 8,196 submissions, 23%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 203Total Downloads
- Downloads (Last 12 months)203
- Downloads (Last 6 weeks)10
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in