skip to main content
research-article

Chinese Emotional Dialogue Response Generation via Reinforcement Learning

Published: 22 July 2021 Publication History

Abstract

In an open-domain dialogue system, recognition and expression of emotions are the key factors for success. Most of the existing research related to Chinese dialogue systems aims at improving the quality of content but ignores the expression of human emotions. In this article, we propose a Chinese emotional dialogue response generation algorithm based on reinforcement learning that can generate responses not only according to content but also according to emotion. In the proposed method, a multi-emotion classification model is first used to add emotion labels to the corpus of post-response pairs. Then, with the help of reinforcement learning, the reward function is constructed based on two aspects, namely, emotion and content. Among the generated candidates, the system selects the one with long-term success as the best reply. At the same time, to avoid safe responses and diversify dialogue, a diversity beam search algorithm is applied in the decoding process. The comparative experiments demonstrate that the proposed model achieves satisfactory results according to both automatic and human evaluations.

References

[1]
Layla El Asri, Jing He, and Kaheer Suleman. 2016. A sequence-to-sequence model for user simulation in spoken dialogue systems. In Interspeech 2016. 1151–1155.
[2]
Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations (ICLR’15).
[3]
Z. Cai, X. Zheng, and J. Yu. 2019. A differential-private framework for urban traffic flows estimation via taxi companies. IEEE Transactions on Industrial Informatics 15, 12 (2019), 6492–6499.
[4]
Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2, 3 (May 2011), Article 27, 27 pages.
[5]
Songsak Channarukul, Susan W. McRoy, and Syed S. Ali. 2001. YAG: A template-based text realization system for dialog. International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems 9, 06 (2001), 649–659.
[6]
Yuling Chen and Zhi Zhang. 2018. Research on text sentiment analysis based on CNNs and SVM. In 2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA’18). IEEE, 2731–2734.
[7]
Zhenrong Deng, Fuxin Ma, Rushi Lan, Wenming Huang, and Xiaonan Luo. 2020. A two-stage Chinese text summarization algorithm using keyword information and adversarial learning. Neurocomputing 425 (2020), 117–126.
[8]
Markus Freitag and Yaser Al-Onaizan. 2017. Beam search strategies for neural machine translation. In Proceedings of the 1st Workshop on Neural Machine Translation. 56–60.
[9]
Daniel Guo, Gokhan Tur, Wen-tau Yih, and Geoffrey Zweig. 2014. Joint semantic utterance classification and slot filling with recursive neural networks. In 2014 IEEE Spoken Language Technology Workshop (SLT’14). IEEE, 554–559.
[10]
Hongyu Guo. 2015. Generating text with deep reinforcement learning. CoRR abs/1510.09202 (2015). arxiv:1510.09202.http://arxiv.org/abs/1510.09202
[11]
Matthew Henderson, Blaise Thomson, and Steve Young. 2013. Deep neural network approach for the dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference. 467–471.
[12]
R. Lan, L. Sun, Z. Liu, H. Lu, C. Pang, and X. Luo. 2020a. MADNet: A fast and lightweight network for single-image super resolution. IEEE Transactions on Cybernetics 51 (2021), 1443–1453.
[13]
R. Lan, L. Sun, Z. Liu, H. Lu, Z. Su, C. Pang, and X. Luo. 2020b. Cascading and enhanced residual networks for accurate single-image super-resolution. IEEE Transactions on Cybernetics 51, 1 (2021), 115–121.
[14]
R. Lan, Y. Zhou, Z. Liu, and X. Luo. 2020c. Prior knowledge-based probabilistic collaborative representation for visual recognition. IEEE Transactions on Cybernetics 50, 4 (2020), 1498–1508.
[15]
Cheongjae Lee, Sangkeun Jung, Seokhwan Kim, and Gary Geunbae Lee. 2009. Example-based dialog modeling for practical multi-domain dialog system. Speech Communication 51, 5 (2009), 466–484.
[16]
H. Li, C. Pun, F. Xu, L. Pan, R. Zong, H. Gao, and H. Lu. 2020. A hybrid feature selection algorithm based on a discrete artificial bee colony for Parkinson’s diagnosis. ACM Transactions on Internet Technology (2020).
[17]
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016a. A simple, fast diverse decoding algorithm for neural generation. arXiv:1611.08562.
[18]
Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016b. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 1192–1202.
[19]
Jing Li, Yan Song, Haisong Zhang, and Shuming Shi. 2018. A manually annotated Chinese corpus for non-task-oriented dialogue systems. arXiv:1805.05542.
[20]
Shudong Li, Zhou Yan, Xiaobo Wu, Aiping Li, and Bin Zhou. 2017. A method of emotional analysis of movie based on convolution neural network and bi-directional LSTM RNN. In 2017 IEEE S2nd International Conference on Data Science in Cyberspace (DSC’17). IEEE, 156–161.
[21]
Xiang Li, Lili Mou, Rui Yan, and Ming Zhang. 2016c. Stalematebreaker: A proactive content-introducing approach to automatic human-computer conversation. In Proceedings of the 25th International Joint Conference on Artificial Intelligence. AAAI Press, 2845–2851.
[22]
Zhongyang Li, Xiao Ding, and Ting Liu. 2018. Generating reasonable and diversified story ending using sequence to sequence model with adversarial training. In Proceedings of the 27th International Conference on Computational Linguistics. 1033–1043.
[23]
Fuhua Lin and Douglas H. Norrie. 2001. Schema-based conversation modeling for agent-oriented manufacturing systems. Computers in Industry 46, 3 (2001), 259–274.
[24]
Chia-Wei Liu, Ryan Lowe, Iulian V. Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2122–2132.
[25]
Weibo Liu, Zidong Wang, Xiaohui Liu, Nianyin Zeng, Yurong Liu, and Fuad E. Alsaadi. 2017. A survey of deep neural network architectures and their applications. Neurocomputing 234 (2017), 11–26.
[26]
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, 1064–1074.
[27]
Hongyuan Mei, Mohit Bansal, and Matthew R. Walter. 2017. Coherent dialogue with attention-based language models. In Proceedings of the 31st AAAI Conference on Artificial Intelligence. AAAI Press, 3252–3258.
[28]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv:1312.5602.
[29]
Manel Mrabet, Yosra Ben Saied, and Leila Azouz Saidane. 2019. CAN-TM: Chain augmented Naïve Bayes-based trust model for reliable cloud service selection. ACM Transactions on Internet Technology 19, 4 (Sept. 2019), Article 47, 20 pages.
[30]
Tong Niu and Mohit Bansal. 2018. Polite dialogue generation without parallel data. Transactions of the Association of Computational Linguistics 6 (2018), 373–389.
[31]
H. Peng, E. Cambria, and A. Hussain. 2017. A review of sentiment analysis research in Chinese language. Cognitive Computation 9, 8 (2017), 423–435.
[32]
Alan Ritter, Colin Cherry, and William B. Dolan. 2011. Data-driven response generation in social media. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 583–593.
[33]
Iulian Vlad Serban, Tim Klinger, Gerald Tesauro, Kartik Talamadupula, Bowen Zhou, Yoshua Bengio, and Aaron Courville. 2017. Multiresolution recurrent neural networks: An application to dialogue response generation. In Proceedings of the 31st AAAI Conference on Artificial Intelligence. AAAI Press, 3288–3294.
[34]
Iulian V. Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the 30h AAAI Conference on Artificial Intelligence. AAAI Press, 3776–3783.
[35]
Amanda Stent, Rashmi Prasad, and Marilyn Walker. 2004. Trainable sentence planning for complex information presentation in spoken dialog systems. In Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, 79.
[36]
David Suendermann, Keelan Evanini, Jackson Liscombe, Phillip Hunter, Krishna Dayanidhi, and Roberto Pieraccini. 2009. From rule-based to statistical grammars: Continuous improvement of large-scale spoken dialog systems. In 2009 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 4713–4716.
[37]
Xiao Sun, Chengcheng Li, and Fuji Ren. 2016. Sentiment analysis for Chinese microblog based on deep neural networks with convolutional extension features. Neurocomputing 210 (2016), 227–236.
[38]
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems. 3104–3112.
[39]
Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2018. Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems. In 32nd AAAI Conference on Artificial Intelligence.
[40]
Zhiyang Teng, Duy Tin Vo, and Yue Zhang. 2016. Context-sensitive lexicon features for neural sentiment analysis. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 1629–1638.
[41]
Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. In 32nd AAAI Conference on Artificial Intelligence. 7371–7379.
[42]
Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv:1506.05869.
[43]
Y. Wang, Y. Gao, S. Li, and X. Tong. 2020. A worker-selection incentive mechanism for optimizing platform-centric mobile crowdsourcing systems. Computer Networks 107 (2020), 107144.
[44]
Zhuoran Wang, Hongliang Chen, Guanchun Wang, Hao Tian, Hua Wu, and Haifeng Wang. 2014. Policy learning for domain selection in an extensible multi-domain spoken dialogue system. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP’14). 57–67.
[45]
Jason Williams, Antoine Raux, Deepak Ramachandran, and Alan Black. 2013. The dialog state tracking challenge. In Proceedings of the SIGDIAL 2013 Conference. 404–413.
[46]
Sam Wiseman and Alexander M. Rush. 2016. Sequence-to-sequence learning as beam-search optimization. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 1296–1306.
[47]
Hong Xie, Yongkun Li, and John C. S. Lui. 2020. A reinforcement learning approach to optimize discount and reputation tradeoffs in e-commerce systems. ACM Transactions on Internet Technology 20, 4 (Oct. 2020), Article 37, 26 pages.
[48]
Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. 2017. Topic aware neural response generation. In Proceedings of the 31st AAAI Conference on Artificial Intelligence. AAAI Press, 3351–3357.
[49]
Hao Zhou, Minlie Huang, Tianyang Zhang, Xiaoyan Zhu, and Bing Liu. Emotional chatting machine: Emotional conversation generation with internal and external memory. In 32nd AAAI Conference on Artificial Intelligence. 730–738.
[50]
Yucan Zhou, Qinghua Hu, Jie Liu, and Yuan Jia. 2015. Combining heterogeneous deep neural networks with conditional random fields for Chinese dialogue act recognition. Neurocomputing 168 (2015), 408–417.

Cited By

View all
  • (2025)CoMaSa:Context Multi-aware Self-attention for emotional response generationNeurocomputing10.1016/j.neucom.2024.128692611(128692)Online publication date: Jan-2025
  • (2024)RLCA: Reinforcement Learning Model Integrating Cognition and Affection for Empathetic Response GenerationIEEE Transactions on Computational Social Systems10.1109/TCSS.2023.325874111:1(1158-1168)Online publication date: Feb-2024
  • (2023)SPK-CG: Siamese Network based Posterior Knowledge Selection Model for Knowledge Driven Conversation GenerationACM Transactions on Asian and Low-Resource Language Information Processing10.1145/356957922:3(1-16)Online publication date: 10-Mar-2023
  • Show More Cited By

Index Terms

  1. Chinese Emotional Dialogue Response Generation via Reinforcement Learning

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Internet Technology
    ACM Transactions on Internet Technology  Volume 21, Issue 4
    November 2021
    520 pages
    ISSN:1533-5399
    EISSN:1557-6051
    DOI:10.1145/3472282
    • Editor:
    • Ling Lu
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 22 July 2021
    Accepted: 01 December 2020
    Revised: 01 November 2020
    Received: 01 August 2020
    Published in TOIT Volume 21, Issue 4

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. Dialogue generation
    2. emotion classification
    3. reinforcement learning
    4. safe response
    5. reward function

    Qualifiers

    • Research-article
    • Refereed

    Funding Sources

    • National Key Research and Development Program of China
    • National Natural Science Foundation of China
    • Guangxi Science and Technology Project
    • Guangxi Key Laboratory of Image and Graphic Intelligent Processing

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)24
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 12 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)CoMaSa:Context Multi-aware Self-attention for emotional response generationNeurocomputing10.1016/j.neucom.2024.128692611(128692)Online publication date: Jan-2025
    • (2024)RLCA: Reinforcement Learning Model Integrating Cognition and Affection for Empathetic Response GenerationIEEE Transactions on Computational Social Systems10.1109/TCSS.2023.325874111:1(1158-1168)Online publication date: Feb-2024
    • (2023)SPK-CG: Siamese Network based Posterior Knowledge Selection Model for Knowledge Driven Conversation GenerationACM Transactions on Asian and Low-Resource Language Information Processing10.1145/356957922:3(1-16)Online publication date: 10-Mar-2023
    • (2023)An Overview of Affective Speech Synthesis and Conversion in the Deep Learning EraProceedings of the IEEE10.1109/JPROC.2023.3250266111:10(1355-1381)Online publication date: Oct-2023
    • (2023)Authentic Dialogue Generation to Improve Youth’s Awareness of Cybergrooming for Online Safety2023 IEEE 35th International Conference on Tools with Artificial Intelligence (ICTAI)10.1109/ICTAI59109.2023.00017(64-69)Online publication date: 6-Nov-2023

    View Options

    Login options

    Full Access

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media