Skip to main content
SpringerLink
Log in

Variational Attention for Commonsense Knowledge Aware Conversation Generation

  • Conference paper
  • First Online:
Natural Language Processing and Chinese Computing (NLPCC 2019)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 11838))

Abstract

Conversation generation is an important task in natural language processing, and commonsense knowledge is vital to provide a shared background for better replying. In this paper, we present a novel commonsense knowledge aware conversation generation model, which adopts variational attention for incorporating commonsense knowledge to generate more appropriate conversation. Given a post, the model retrieves relevant knowledge graphs from a knowledge base, and then attentively incorporates knowledge to its response. For enhancing attention to incorporate more clean and suitable knowledge into response generation, we adopt variational attention rather than standard neural attention on knowledge graphs, which is unlike previous knowledge aware generation models. Experimental results show that the variational attention based model can incorporate more clean and suitable knowledge into response generation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://conceptnet.io.

  2. 2.

    https://www.reddit.com/r/datasets/comments/3bxlg7/i_have_every_publicly_available_reddit_comment/.

References

  1. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)

  2. Bahuleyan, H., Mou, L., Vechtomova, O., Poupart, P.: Variational attention for sequence-to-sequence models. arXiv preprint arXiv:1712.08207 (2017)

  3. Bordes, A., Usunier, N., Garcia-Duran, A., Weston, J., Yakhnenko, O.: Translating embeddings for modeling multi-relational data. In: Advances in Neural Information Processing Systems, pp. 2787–2795 (2013)

    Google Scholar 

  4. Bowman, S.R., Vilnis, L., Vinyals, O., Dai, A.M., Jozefowicz, R., Bengio, S.: Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349 (2015)

  5. Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)

  6. Deng, Y., Kim, Y., Chiu, J., Guo, D., Rush, A.M.: Latent alignment and variational attention. arXiv preprint arXiv:1807.03756 (2018)

  7. Ghazvininejad, M., et al.: A knowledge-grounded neural conversation model. arXiv preprint arXiv:1702.01932 (2017)

  8. Gu, J., Lu, Z., Li, H., Li, V.O.: Incorporating copying mechanism in sequence-to-sequence learning. arXiv preprint arXiv:1603.06393 (2016)

  9. He, S., Liu, C., Liu, K., Zhao, J.: Generating natural answers by incorporating copying and retrieving mechanisms in sequence-to-sequence learning. In: ACL, vol. 1, pp. 199–208 (2017)

    Google Scholar 

  10. Kingma, D.P., Welling, M.: Auto-Encoding Variational Bayes (2013)

    Google Scholar 

  11. Li, J., Galley, M., Brockett, C., Gao, J., Dolan, B.: A diversity-promoting objective function for neural conversation models. In: NAACL, pp. 110–119 (2016)

    Google Scholar 

  12. Li, J., Monroe, W., Ritter, A., Jurafsky, D., Galley, M., Gao, J.: Deep reinforcement learning for dialogue generation. In: EMNLP, pp. 1192–1202 (2016)

    Google Scholar 

  13. Long, Y., Wang, J., Xu, Z., Wang, Z., Wang, B., Wang, Z.: A knowledge enhanced generative conversational service agent. In: Proceedings of the 6th Dialog System Technology Challenges (DSTC6) Workshop (2017)

    Google Scholar 

  14. Lyu, C., Titov, I.: AMR parsing as graph prediction with latent alignment. In: ACL, pp. 397–407 (2018)

    Google Scholar 

  15. Marková, I., Linell, P., Grossen, M., Salazar Orvig, A.: Dialogue in Focus Groups: Exploring Socially Shared Knowledge. Equinox Publishing (2007)

    Google Scholar 

  16. Minsky, M.: Society of mind: a response to four reviews. Artif. Intell. 48(3), 371–396 (1991)

    Article  MathSciNet  Google Scholar 

  17. Mou, L., Song, Y., Yan, R., Li, G., Zhang, L., Jin, Z.: Sequence to backward and forward sequences: a content-introducing approach to generative short-text conversation. arXiv preprint arXiv:1607.00970 (2016)

  18. do Nascimento Souto, P.C.: Creating knowledge with and from the differences: the required dialogicality and dialogical competences. RAI-Revista de Administração e Inovação 12(2), 60–89 (2015)

    Google Scholar 

  19. Serban, I.V., Sordoni, A., Bengio, Y., Courville, A.C., Pineau, J.: Building end-to-end dialogue systems using generative hierarchical neural network models. In: AAAI, vol. 16, pp. 3776–3784 (2016)

    Google Scholar 

  20. Serban, I.V., et al.: A hierarchical latent variable encoder-decoder model for generating dialogues. In: AAAI, pp. 3295–3301 (2017)

    Google Scholar 

  21. Shang, L., Lu, Z., Li, H.: Neural responding machine for short-text conversation. arXiv preprint arXiv:1503.02364 (2015)

  22. Sordoni, A., Galley, M., Auli, M., Brockett, C., Ji, Y., Mitchell, M., Nie, J.Y., Gao, J., Dolan, B.: A neural network approach to context-sensitive generation of conversational responses. In: NAACL, pp. 196–205 (2015)

    Google Scholar 

  23. Speer, R., Havasi, C.: Representing general relational knowledge in ConceptNet 5. In: LREC, pp. 3679–3686 (2012)

    Google Scholar 

  24. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112 (2014)

    Google Scholar 

  25. Xing, C., Wu, W., Wu, Y., Liu, J., Huang, Y., Zhou, M., Ma, W.Y.: Topic aware neural response generation. In: AAAI, vol. 17, pp. 3351–3357 (2017)

    Google Scholar 

  26. Xu, Z., Liu, B., Wang, B., Sun, C., Wang, X.: Incorporating loose-structured knowledge into conversation modeling via recall-gate LSTM. In: 2017 International Joint Conference on Neural Networks (IJCNN), pp. 3506–3513. IEEE (2017)

    Google Scholar 

  27. Zhang, R., Guo, J., Fan, Y., Lan, Y., Xu, J., Cheng, X.: Learning to control the specificity in neural response generation. In: ACL, pp. 1108–1117 (2018)

    Google Scholar 

  28. Zhao, T., Zhao, R., Eskenazi, M.: Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. In: ACL, pp. 654–664 (2017)

    Google Scholar 

  29. Zhou, H., Young, T., Huang, M., Zhao, H., Xu, J., Zhu, X.: Commonsense knowledge aware conversation generation with graph attention. In: IJCAI, pp. 4623–4629 (2018)

    Google Scholar 

  30. Zhu, W., Mo, K., Zhang, Y., Zhu, Z., Peng, X., Yang, Q.: Flexible end-to-end dialogue system for knowledge grounded conversation. arXiv preprint arXiv:1709.04264 (2017)

Download references

Acknowledgement

This work is supported by the National Natural Science Foundation of China (No. 61533018), the Natural Key R&D Program of China (No. 2018YFC0830101), the National Natural Science Foundation of China (No. 61702512, No. 61806201) and the independent research project of National Laboratory of Pattern Recognition. This work was also supported by CCF-DiDi BigData Joint Lab and CCF-Tencent Open Research Fund.

Author information

Authors and Affiliations

  1. National Laboratory of Pattern Recognition Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China

    Guirong Bai, Shizhu He, Kang Liu & Jun Zhao

  2. University of Chinese Academy of Sciences, Beijing, 100049, China

    Guirong Bai, Kang Liu & Jun Zhao

Authors
  1. Guirong Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shizhu He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jun Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jun Zhao .

Editor information

Editors and Affiliations

  1. Tsinghua University, Beijing, China

    Jie Tang

  2. National University of Singapore, Singapore, Singapore

    Min-Yen Kan

  3. Peking University, Beijing, China

    Dongyan Zhao

  4. Peking University, Beijing, China

    Sujian Li

  5. Zhengzhou University, Zhengzhou, China

    Hongying Zan

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bai, G., He, S., Liu, K., Zhao, J. (2019). Variational Attention for Commonsense Knowledge Aware Conversation Generation. In: Tang, J., Kan, MY., Zhao, D., Li, S., Zan, H. (eds) Natural Language Processing and Chinese Computing. NLPCC 2019. Lecture Notes in Computer Science(), vol 11838. Springer, Cham. https://doi.org/10.1007/978-3-030-32233-5_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-32233-5_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-32232-8

  • Online ISBN: 978-3-030-32233-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation