skip to main content
10.1145/3357384.3358048acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
research-article

ConCET: Entity-Aware Topic Classification for Open-Domain Conversational Agents

Published: 03 November 2019 Publication History

Abstract

Identifying the topic (domain) of each user's utterance in open-domain conversational systems is a crucial step for all subsequent language understanding and response tasks. In particular, for complex domains, an utterance is often routed to a single component responsible for that domain. Thus, correctly mapping a user utterance to the right domain is critical. This is a challenging task: users could mention entities like actors, singers or locations to implicitly indicate the domain, which requires extensive domain knowledge to interpret. To address this problem, we introduce ConCET: a Concurrent Entity-aware conversational Topic classifier, which incorporates entity type information together with the utterance content features. Specifically, ConCET utilizes entity information to enrich the utterance representation, combining character, word, and entity type embeddings into a single representation. However, for rich domains with millions of available entities, unrealistic amounts of labeled training data would be required. To complement our model, we propose a simple and effective method for generating synthetic training data, to augment the typically limited amounts of labeled training data, using commonly available knowledge bases as to generate additional labeled utterances. We extensively evaluate ConCET and our proposed training method first on an openly available human-human conversational dataset called Self-Dialogue, to calibrate our approach against previous state-of-the-art methods; second, we evaluate ConCET on a large dataset of human-machine conversations with real users, collected as part of the Amazon Alexa Prize. Our results show that ConCET significantly improves topic classification performance on both datasets, reaching 8-10% improvements compared to state-of-the-art deep learning methods. We complement our quantitative results with detailed analysis of system performance, which could be used for further improvements of conversational agents.

References

[1]
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In The semantic web. Springer, 722--735.
[2]
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. (2017), 135--146.
[3]
Gerlof Bouma. 2009. Normalized (pointwise) mutual information in collocation extraction. Proceedings of GSCL (2009), 31--40.
[4]
Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, and et al. 2018. Universal sentence encoder. In arXiv preprint arXiv:1803.11175.
[5]
Alexis Conneau, Holger Schwenk, Loc Barrault, and Yann Lecun. 2017. Very deep convolutional networks for text classification. In 15th Conference of the European Chapter of the Association for Computational Linguistics(EACL).
[6]
Marco Cornolti, Paolo Ferragina, Massimiliano Ciaramita, Stefan Rüd, and Hinrich Schütze. 2016. A piggyback system for joint entity mention detection and linking in web queries. In Proceedings of the 25th International Conference on World Wide Web. 567--578.
[7]
Fenfei Guo, Angeliki Metallinou, Chandra Khatri, Anirudh Raju, Anu Venkatesh, and Ashwin Ram. 2018. Topic-based evaluation for conversational bots. In NIPS.
[8]
Joty, Muhammad Tasnim Mohiuddin, and Dat Tien Nguyen. 2018. Coherence modeling of asynchronous conversations: A neural entity grid approach. In proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 558--568.
[9]
Shafiq Joty, Muhammad Tasnim Mohiuddin, and Dat Tien Nguyen. 2018. Coherence modeling of asynchronous conversations: A neural entity grid approach. In proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 558--568.
[10]
Chandra Khatri, Rahul Goel, Behnam Hedayatnia, Angeliki Metanillou, Anushree Venkatesh, Raefer Gabriel, and Arindam Mandal. 2018. Contextual Topic Modeling For Dialog Systems. In IEEE 2018 Spoken Language Technology (SLT). IEEE.
[11]
Y. Kim. 2014. Convolutional neural networks for sentence classification. In proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).
[12]
Ben Krause, Marco Damonte, Mihai Dobre, Daniel Duma, Joachim Fainberg, Federico Fancellu, Emmanuel Kahembwe, Jianpeng Cheng, and Bonnie Webber. 2017. Edina: Building an open domain socialbot with self-dialogues. In 1st Proceedings of the Alexa Prize. Amazon.
[13]
Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. In proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI).
[14]
Pablo N Mendes, Max Jakob, Andrés García-Silva, and Christian Bizer. 2011. DBpedia spotlight: shedding light on the web of documents. In Proceedings of the 7th international conference on semantic systems. ACM, 1--8.
[15]
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In arXiv preprint arXiv:1301.3781.
[16]
David Mimno, and Edmund Talley Hanna M. Wallach, Miriam Leenders, and Andrew McCallum. 2011. Optimizing semantic coherence in topic models. In proceedings of the conference on empirical methods in natural language processing. Association for Computational Linguistics, 262--272.
[17]
Tom M Mitchell. 1997. Machine learning. Number 35. Burr Ridge, IL: McGraw Hill 45.
[18]
Andrea Moro, Alessandro Raganato, and Roberto Navigli. 2014. Entity linking meets word sense disambiguation: a unified approach. Transactions of the Association for Computational Linguistics 2 (2014), 231--244.
[19]
Matt Post and Shane Bergsma. 2013. Explicit and implicit syntactic features for text classification. In proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. 866--872.
[20]
Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, and Jeff Nunn. 2018. Conversational AI: The science behind the alexa prize. In 1st Proceedings of Alexa Prize. Amazon.
[21]
Anu Venkatesh, Chandra Khatri, Ashwin Ram, Fenfei Guo, Raefer Gabriel, Ashish Nagar, Rohit Prasad, Ming Cheng, Behnam Hedayatnia, Angeliki Metallinou, Rahul Goel, Shaohua Yang, and Anirudh Raju. 2018. On Evaluating and Comparing Conversational Agents. In NIPS.
[22]
JinWang, ZhongyuanWang, Dawei Zhang, and Jun Yan. 2017. Combining Knowledge with Deep Convolutional Neural Networks for Short Text Classification. In IJCAI. 2915--2921.
[23]
Zhongyuan Wang and Haixun Wang. 2016. Understanding short texts(tutorial). In ACL.
[24]
Chenyan Xiong, Jamie Callan, and Tie-Yan Liu. 2016. Bag-of-Entities representation for ranking. In Proceedings of the ICTIR, 181--184.
[25]
Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2017. Learning distributed representations of texts and entities from knowledge base. Transactions of the Association for Computational Linguistics, 397--411.
[26]
Jui-Feng Yeh, Chen-Hsien Lee, Yi-Shiuan Tan, and Liang-Chih Yu. 2014. Topic model allocation of conversational dialogue records by Latent Dirichlet Allocation. In In Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2014 Asia-Pacific. IEEE, 1--4.

Cited By

View all
  • (2023)Cinnalyst: A Sales Meeting Analysis System2023 15th International Conference on Knowledge and Systems Engineering (KSE)10.1109/KSE59128.2023.10299492(1-6)Online publication date: 18-Oct-2023
  • (2022)Dealing With Hierarchical Types and Label Noise in Fine-Grained Entity TypingIEEE/ACM Transactions on Audio, Speech, and Language Processing10.1109/TASLP.2022.315528130(1305-1318)Online publication date: 2022
  • (2021)Report on the WSDM 2020 workshop on state-based user modelling (SUM'20)ACM SIGIR Forum10.1145/3451964.345196954:1(1-11)Online publication date: 19-Feb-2021
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CIKM '19: Proceedings of the 28th ACM International Conference on Information and Knowledge Management
November 2019
3373 pages
ISBN:9781450369763
DOI:10.1145/3357384
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2019

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. conversational topic classification
  2. entity-aware conversation domain classification
  3. open-domain conversational agents

Qualifiers

  • Research-article

Funding Sources

  • Amazon Alexa

Conference

CIKM '19
Sponsor:

Acceptance Rates

CIKM '19 Paper Acceptance Rate 202 of 1,031 submissions, 20%;
Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)14
  • Downloads (Last 6 weeks)0
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)Cinnalyst: A Sales Meeting Analysis System2023 15th International Conference on Knowledge and Systems Engineering (KSE)10.1109/KSE59128.2023.10299492(1-6)Online publication date: 18-Oct-2023
  • (2022)Dealing With Hierarchical Types and Label Noise in Fine-Grained Entity TypingIEEE/ACM Transactions on Audio, Speech, and Language Processing10.1109/TASLP.2022.315528130(1305-1318)Online publication date: 2022
  • (2021)Report on the WSDM 2020 workshop on state-based user modelling (SUM'20)ACM SIGIR Forum10.1145/3451964.345196954:1(1-11)Online publication date: 19-Feb-2021
  • (2020)User Intent Inference for Web Search and Conversational AgentsProceedings of the 13th International Conference on Web Search and Data Mining10.1145/3336191.3372187(911-912)Online publication date: 20-Jan-2020
  • (2019)Offline and Online Satisfaction Prediction in Open-Domain Conversational SystemsProceedings of the 28th ACM International Conference on Information and Knowledge Management10.1145/3357384.3358047(1281-1290)Online publication date: 3-Nov-2019

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media