Wei Yuan
ABSTRACT
We present a few-shot intent detection model for an enterprise’s conversational dialogue system. The model uses an intent topological tree to guide the search for the user intent using large language models (LLMs). The intents are resolved based on semantic similarities between user utterances and the text descriptions of the internal nodes of the intent tree or the intent examples in the leaf nodes of the tree. Our results show that an off-the-shelf language model can work reasonably well in a large enterprise deployment without fine-tuning, and its performance can be further improved with fine-tuning as more domain-specific data becomes available. We also show that the fine-tuned language model meets and outperforms the state-of-the-art (SOTA) results in resolving conversation intents without training classifiers. With the use of a topological intent tree, our model provides more interpretability to cultivate people’s trust in their decisions.
- Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient Intent Detection with Dual Sentence Encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI. Association for Computational Linguistics, Online, 38–45. https://doi.org/10.18653/v1/2020.nlp4convai-1.5Google Scholar
Cross Ref
- Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Céspedes, Steve Yuan, Chris Tar, 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175 (2018).Google Scholar
- Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple Contrastive Learning of Sentence Embeddings. arXiv preprint arXiv:2104.08821 (2021).Google Scholar
- R. Hadsell, S. Chopra, and Y. LeCun. 2006. Dimensionality Reduction by Learning an Invariant Mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), Vol. 2. 1735–1742. https://doi.org/10.1109/CVPR.2006.100Google ScholarDigital Library
- Matthew Henderson, Iñigo Casanueva, Nikola Mrkšić, Pei-Hao Su, Tsung-Hsien Wen, and Ivan Vulić. 2019. ConveRT: Efficient and accurate conversational representations from transformers. arXiv preprint arXiv:1911.03688 (2019).Google Scholar
- Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). https://www.aclweb.org/anthology/D19-1131Google Scholar
- Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. 2019. Benchmarking natural language understanding services for building conversational agents.Google Scholar
- Shikib Mehri and Mihail Eric. 2021. Example-Driven Intent Prediction with Observers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Online, 2979–2992. https://doi.org/10.18653/v1/2021.naacl-main.237Google ScholarCross Ref
- S. Mehri, M. Eric, and D. Hakkani-Tur. 2020. DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue. ArXiv abs/2009.13570 (2020).Google Scholar
- Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. https://arxiv.org/abs/1908.10084Google ScholarCross Ref
- Jetze Schuurmans and Flavius Frasincar. 2020. Intent Classification for Dialogue Utterances. IEEE Intelligent Systems 35, 1 (2020), 82–88. https://doi.org/10.1109/MIS.2019.2954966Google ScholarDigital Library
- Ivan Vulić, Pei-Hao Su, Samuel Coope, Daniela Gerz, Paweł Budzianowski, Iñigo Casanueva, Nikola Mrkšić, and Tsung-Hsien Wen. 2021. ConvFiT: Conversational Fine-Tuning of Pretrained Language Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 1151–1168. https://doi.org/10.18653/v1/2021.emnlp-main.88Google ScholarCross Ref
- Congying Xia, Wenpeng Yin, Yihao Feng, and Philip Yu. 2021. Incremental few-shot text classification with multi-round new classes: Formulation, dataset and system. arXiv preprint arXiv:2104.11882 (2021).Google Scholar
- Jianguo Zhang, Trung Bui, Seunghyun Yoon, Xiang Chen, Zhiwei Liu, Congying Xia, Quan Hung Tran, Walter Chang, and Philip Yu. 2021. Few-Shot Intent Detection via Contrastive Pre-Training and Fine-Tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 1906–1912. https://doi.org/10.18653/v1/2021.emnlp-main.144Google ScholarCross Ref
- Jianguo Zhang, Kazuma Hashimoto, Wenhao Liu, Chien-Sheng Wu, Yao Wan, Philip Yu, Richard Socher, and Caiming Xiong. 2020. Discriminative Nearest Neighbor Few-Shot Intent Detection by Transferring Natural Language Inference. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Online, 5064–5082. https://doi.org/10.18653/v1/2020.emnlp-main.411Google ScholarCross Ref
- Zheng Zhang, Ryuichi Takanobu, Qi Zhu, MinLie Huang, and XiaoYan Zhu. 2020. Recent advances and challenges in task-oriented dialog systems. Science China Technological Sciences (2020), 1–17.Google Scholar
Index Terms
- Contrastive Fine-tuning on Few Shot Intent Detection with Topological Intent Tree
Recommendations
Characterizing commercial intent
CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge managementUnderstanding the intent underlying user's queries may help personalize search results and therefore improve user satisfaction. We develop a methodology for using the content of search engine result pages (SERPs) along with the information obtained from ...
Characterizing search intent diversity into click models
WWW '11: Proceedings of the 20th international conference on World wide webModeling a user's click-through behavior in click logs is a challenging task due to the well-known position bias problem. Recent advances in click models have adopted the examination hypothesis which distinguishes document relevance from position bias. ...
Quality-biased ranking for queries with commercial intent
WWW '13 Companion: Proceedings of the 22nd International Conference on World Wide WebModern search engines are good enough to answer popular commercial queries with mainly highly relevant documents. However, our experiments show that users behavior on such relevant commercial sites may differ from one to another web-site with the same ...
Comments