research-article

Multi-Turn Response Selection in Retrieval-Based Chatbots with Iterated Attentive Convolution Matching Network

Authors:

Junyu ChenAuthors Info & Claims

CIKM '19: Proceedings of the 28th ACM International Conference on Information and Knowledge Management

Pages 1081 - 1090

https://doi.org/10.1145/3357384.3357928

Published: 03 November 2019 Publication History

Abstract

Building an intelligent chatbot with multi-turn dialogue ability is a major challenge, which requires understanding the multi-view semantic and dependency correlation among words, n-grams and sub-sequences. In this paper, we investigate selecting the proper response for a context through multi-grained representation and interactive matching. To construct hierarchical representation types of text segments, we propose a refined architecture which exclusively consists of gated dilated-convolution and self-attention. Compared with the recurrent-based sentence modeling methods, this architecture provides more flexibility and a speedup. The matching signals of each utterance-response pair are extracted by integrating the interactive information from different views. Then a turns-aware attention mechanism is utilized to aggregate the matching sequence, so as to identify important utterances and capture the implicit relationship of the whole context. Experiments on two large-scale public data sets show that our model significantly outperforms the state-of-the-art methods in terms of all metrics. We empirically provide a thorough ablation test, as well as the comparison of different representation and matching strategies, for a better insight into how each component affects the performance of the model.

References

[1]

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. CoRR, Vol. abs/1607.06450 (2016).

[2]

Ricardo A. Baeza-Yates and Berthier A. Ribeiro-Neto. 1999. Modern Information Retrieval .ACM Press / Addison-Wesley.

Digital Library

[3]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014).

[4]

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, Vol. abs/1412.3555 (2014).

[5]

Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. In ICML (Proceedings of Machine Learning Research), Vol. 70. PMLR, 933--941.

Digital Library

[6]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. IEEE Computer Society, 770--778.

[7]

Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014. Convolutional neural network architectures for matching natural language sentences. In Advances in neural information processing systems. 2042--2050.

[8]

Zongcheng Ji, Zhengdong Lu, and Hang Li. 2014. An information retrieval approach to short text conversation. arXiv preprint arXiv:1408.6988 (2014).

[9]

Rudolf Kadlec, Martin Schmid, and Jan Kleindienst. 2015. Improved deep learning baselines for ubuntu corpus dialogs. arXiv preprint arXiv:1510.03753 (2015).

[10]

Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.

[11]

Feng-Lin Li, Minghui Qiu, Haiqing Chen, Xiongwei Wang, Xing Gao, Jun Huang, Juwei Ren, Zhongzhou Zhao, Weipeng Zhao, Lei Wang, Guwei Jin, and Wei Chu. 2017b. AliMe Assist : An Intelligent Assistant for Creating an Innovative E-commerce Experience. In CIKM. ACM, 2495--2498.

[12]

Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016. Deep reinforcement learning for dialogue generation. arXiv preprint arXiv:1606.01541 (2016).

[13]

Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. 2017a. Adversarial learning for neural dialogue generation. arXiv preprint arXiv:1701.06547 (2017).

[14]

Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In SIGDIAL Conference. The Association for Computer Linguistics, 285--294.

[15]

Ryan Thomas Lowe, Nissan Pow, Iulian Vlad Serban, Laurent Charlin, Chia-Wei Liu, and Joelle Pineau. 2017. Training end-to-end dialogue systems with the ubuntu dialogue corpus. Dialogue & Discourse, Vol. 8, 1 (2017), 31--65.

[16]

Zhengdong Lu and Hang Li. 2013. A deep architecture for matching short texts. In Advances in neural information processing systems. 1367--1375.

[17]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS. 3111--3119.

[18]

Lili Mou, Yiping Song, Rui Yan, Ge Li, Lu Zhang, and Zhi Jin. 2016. Sequence to backward and forward sequences: A content-introducing approach to generative short-text conversation. arXiv preprint arXiv:1607.00970 (2016).

[19]

Iulian Vlad Serban, Tim Klinger, Gerald Tesauro, Kartik Talamadupula, Bowen Zhou, Yoshua Bengio, and Aaron Courville. 2017a. Multiresolution recurrent neural networks: An application to dialogue response generation. In AAAI. AAAI Press, 3288--3294.

[20]

Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Thirtieth AAAI Conference on Artificial Intelligence .

Digital Library

[21]

Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017b. A hierarchical latent variable encoder-decoder model for generating dialogues. In AAAI. AAAI Press, 3295--3301.

[22]

Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. arXiv preprint arXiv:1503.02364 (2015).

[23]

Heung-Yeung Shum, Xiaodong He, and Di Li. 2018. From Eliza to XiaoIce: challenges and opportunities with social chatbots. Frontiers of IT & EE, Vol. 19, 1 (2018), 10--26.

[24]

Yan Song, Shuming Shi, Jing Li, and Haisong Zhang. 2018. Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings. In NAACL-HLT (2). Association for Computational Linguistics, 175--180.

[25]

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, Vol. 15, 1 (2014), 1929--1958.

Digital Library

[26]

Emma Strubell, Patrick Verga, David Belanger, and Andrew McCallum. 2017. Fast and accurate entity recognition with iterated dilated convolutions. In EMNLP. Association for Computational Linguistics, 2670--2680.

[27]

Ming Tan, Cicero dos Santos, Bing Xiang, and Bowen Zhou. 2015. LSTM-based deep learning models for non-factoid answer selection. CoRR, Vol. abs/1511.04108 (2015).

[28]

A Turing. 1950. Computing machinery and intelligence. Mind LIX (236): 433--460. Reprinted as (1950), 40--66.

[29]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. 6000--6010.

[30]

Ellen M Voorhees et almbox. 1999. The TREC-8 question answering track report. In Trec, Vol. 99. Citeseer, 77--82.

[31]

Shengxian Wan, Yanyan Lan, Jun Xu, Jiafeng Guo, Liang Pang, and Xueqi Cheng. 2016. Match-srnn: Modeling the recursive matching structure with spatial rnn. In IJCAI. IJCAI/AAAI Press, 2922--2928.

[32]

Hao Wang, Zhengdong Lu, Hang Li, and Enhong Chen. 2013. A dataset for research on short-text conversations. In EMNLP. ACL, 935--945.

[33]

Shuohang Wang and Jing Jiang. 2016. Learning natural language inference with LSTM. In HLT-NAACL. The Association for Computational Linguistics, 1442--1451.

[34]

Felix Wu, Ni Lao, John Blitzer, Guandao Yang, and Kilian Q. Weinberger. 2017a. Fast Reading Comprehension with ConvNets. CoRR, Vol. abs/1711.04352 (2017).

[35]

Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. 2017b. Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. In ACL (1). Association for Computational Linguistics, 496--505.

[36]

Yu Wu, Wei Wu, Dejian Yang, Can Xu, and Zhoujun Li. 2018. Neural response generation with dynamic vocabularies. In AAAI. AAAI Press, 5594--5601.

[37]

Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. 2017. Topic aware neural response generation. In Thirty-First AAAI Conference on Artificial Intelligence .

Digital Library

[38]

Zhen Xu, Bingquan Liu, Baoxun Wang, SUN Chengjie, Xiaolong Wang, Zhuoran Wang, and Chao Qi. 2017. Neural response generation via gan with an approximate embedding layer. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing . 617--626.

[39]

Zhen Xu, Bingquan Liu, Baoxun Wang, Chengjie Sun, and Xiaolong Wang. 2016. Incorporating loose-structured knowledge into lstm with recall gate for conversation modeling. CoRR, Vol. abs/1605.05110 (2016).

[40]

Rui Yan, Yiping Song, and Hua Wu. 2016. Learning to respond with deep neural networks for retrieval-based human-computer conversation system. In SIGIR. ACM, 55--64.

[41]

Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In HLT-NAACL. The Association for Computational Linguistics, 1480--1489.

[42]

Fisher Yu and Vladlen Koltun. 2016. Multi-Scale Context Aggregation by Dilated Convolutions. In ICLR.

[43]

Zhuosheng Zhang, Jiangtong Li, Pengfei Zhu, Hai Zhao, and Gongshen Liu. 2018. Modeling multi-turn conversation with deep utterance aggregation. In COLING. Association for Computational Linguistics, 3740--3752.

[44]

Xiangyang Zhou, Daxiang Dong, Hua Wu, Shiqi Zhao, Dianhai Yu, Hao Tian, Xuan Liu, and Rui Yan. 2016. Multi-view response selection for human-computer conversation. In EMNLP. The Association for Computational Linguistics, 372--381.

[45]

Xiangyang Zhou, Lu Li, Daxiang Dong, Yi Liu, Ying Chen, Wayne Xin Zhao, Dianhai Yu, and Hua Wu. 2018. Multi-turn response selection for chatbots with deep attention matching network. In ACL (1). Association for Computational Linguistics, 1118--1127.

Cited By

Ma ZMei YLong YSu ZGajos K(2024)Evaluating the Experience of LGBTQ+ People Using Large Language Model Based Chatbots for Mental Health SupportProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642482(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642482
Wang QWu LLi G(2024)PIRTRE-C: A Two-Stage Retrieval and Reranking Enhanced Framework for Improving Chinese Psychological Counseling2024 IEEE International Conference on Medical Artificial Intelligence (MedAI)10.1109/MedAI62885.2024.00051(337-347)Online publication date: 15-Nov-2024
https://doi.org/10.1109/MedAI62885.2024.00051
Xie YSun CLiu BJi Z(2024)Multi-View Contrastive Parsing Network for Emotion Recognition in Multi-Party Conversations2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650690(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10650690
Show More Cited By

Index Terms

Multi-Turn Response Selection in Retrieval-Based Chatbots with Iterated Attentive Convolution Matching Network
1. Information systems
  1. Information retrieval
    1. Retrieval models and ranking

Recommendations

Speaker-Aware BERT for Multi-Turn Response Selection in Retrieval-Based Chatbots
CIKM '20: Proceedings of the 29th ACM International Conference on Information & Knowledge Management

In this paper, we study the problem of employing pre-trained language models for multi-turn response selection in retrieval-based chatbots. A new model, named Speaker-Aware BERT (SA-BERT), is proposed in order to make the model aware of the speaker ...
Multi-Representation Fusion Network for Multi-Turn Response Selection in Retrieval-Based Chatbots
WSDM '19: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining

We consider context-response matching with multiple types of representations for multi-turn response selection in retrieval-based chatbots. The representations encode semantics of contexts and responses on words, n-grams, and sub-sequences of utterances, ...
Interactive Matching Network for Multi-Turn Response Selection in Retrieval-Based Chatbots
CIKM '19: Proceedings of the 28th ACM International Conference on Information and Knowledge Management

In this paper, we propose an interactive matching network (IMN) for the multi-turn response selection task. First, IMN constructs word representations from three aspects to address the challenge of out-of-vocabulary (OOV) words. Second, an attentive ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '19: Proceedings of the 28th ACM International Conference on Information and Knowledge Management

November 2019

3373 pages

ISBN:9781450369763

DOI:10.1145/3357384

General Chairs:
Wenwu Zhu
Tsinghua University, China
,
Dacheng Tao
University of Massachusetts, USA
,
Xueqi Cheng
Institute of Computing Technology, CAS, China
,
Program Chairs:
Peng Cui
Tsinghua University, China
,
Elke Rundensteiner
Worcester Polytechnic Institute, USA
,
David Carmel
Amazon Research, USA
,
Qi He
LinkedIn, USA
,
Jeffrey Xu Yu
Chinese University of Hong Kong, China

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CIKM '19

Sponsor:

CIKM '19: The 28th ACM International Conference on Information and Knowledge Management

November 3 - 7, 2019

Beijing, China

Acceptance Rates

CIKM '19 Paper Acceptance Rate 202 of 1,031 submissions, 20%;

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
485
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)1

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ma ZMei YLong YSu ZGajos K(2024)Evaluating the Experience of LGBTQ+ People Using Large Language Model Based Chatbots for Mental Health SupportProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642482(1-15)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642482
Wang QWu LLi G(2024)PIRTRE-C: A Two-Stage Retrieval and Reranking Enhanced Framework for Improving Chinese Psychological Counseling2024 IEEE International Conference on Medical Artificial Intelligence (MedAI)10.1109/MedAI62885.2024.00051(337-347)Online publication date: 15-Nov-2024
https://doi.org/10.1109/MedAI62885.2024.00051
Xie YSun CLiu BJi Z(2024)Multi-View Contrastive Parsing Network for Emotion Recognition in Multi-Party Conversations2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650690(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10650690
Askari AAliannejadi MAbolghasemi AKanoulas EVerberne SFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)CLosER: Conversational Legal Longformer with Expertise-Aware Passage Response Ranker for Long ContextsProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3614812(25-35)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3614812
Ding ZElibol AChong N(2022)Leveraging Extended Chat History through Sentence Embedding in Multi-turn Dialogue toward Increasing User Engagement2022 22nd International Conference on Control, Automation and Systems (ICCAS)10.23919/ICCAS55662.2022.10003889(642-649)Online publication date: 27-Nov-2022
https://doi.org/10.23919/ICCAS55662.2022.10003889
Tsai SPeng YLee W(2022)Multi-turn Query with Similarity Feedback Facilitates Multimodal Video Clip Retrieval2022 IEEE Eighth International Conference on Multimedia Big Data (BigMM)10.1109/BigMM55396.2022.00019(79-86)Online publication date: Dec-2022
https://doi.org/10.1109/BigMM55396.2022.00019
Xu RTao CFeng JWu WYan RZhao D(2021)Response Ranking with Multi-types of Deep Interactive Representations in Retrieval-based DialoguesACM Transactions on Information Systems10.1145/346220739:4(1-28)Online publication date: 17-Aug-2021
https://dl.acm.org/doi/10.1145/3462207
Wardhana AFerdiana RHidayah I(2021)Empathetic Chatbot Enhancement and Development: A Literature Review2021 International Conference on Artificial Intelligence and Mechatronics Systems (AIMS)10.1109/AIMS52415.2021.9466027(1-6)Online publication date: 28-Apr-2021
https://doi.org/10.1109/AIMS52415.2021.9466027
Ma HWang JLin HZhang Y(2021)Hierarchical matching network for multi-turn response selection in retrieval-based chatbotsSoft Computing10.1007/s00500-021-05699-0Online publication date: 29-Mar-2021
https://doi.org/10.1007/s00500-021-05699-0

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten