Abstract
In this paper, we propose a goal-oriented knowledge-driven neural dialog generation system, which leads the conversation based on a knowledge graph. During the conversation, the system has to actively integrate appropriate knowledge conditioned on current dialog state, and then generate coherent, fluent and meaningful responses. We use ERNIE as our backbone model, proposing a fine-tuning scheme to first pre-train on knowledge graph and dialog sequence, and then fine-tune to generate the next response. We extend multi-task learning in multi-turn dialog generation to improve consistency. We show that with well-designed transfer learning, ERNIE shows competitive performance on a knowledge-grounded dialog generation task. In the Baidu knowledge-driven dialog competition, our best single model achieved 4th in the automatic evaluation stage with 47.03 f1 score and 0.417/0.281 BLEU1/BLEU2 score, and ranked 1st in the final human evaluation stage, with descent topic completion performance(1.81/3) and highest coherence score(2.59/3).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Dai, Z., Yang, Z., Yang, Y., Cohen, W.W., Carbonell, J., Le, Q.V., Salakhutdinov, R.: Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860 (2019)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Dinan, E., et al.: The second conversational intelligence challenge (convai2). arXiv preprint arXiv:1902.00098 (2019)
Dinan, E., Roller, S., Shuster, K., Fan, A., Auli, M., Weston, J.: Wizard of wikipedia: Knowledge-powered conversational agents. arXiv preprint arXiv:1811.01241 (2018)
Dong, L., et al.: Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03197 (2019)
Li, J., Galley, M., Brockett, C., Gao, J., Dolan, B.: A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055 (2015)
Li, J., Galley, M., Brockett, C., Spithourakis, G.P., Gao, J., Dolan, B.: A persona-based neural conversation model. arXiv preprint arXiv:1603.06155 (2016)
Liu, S., Chen, H., Ren, Z., Feng, Y., Liu, Q., Yin, D.: Knowledge diffusion for neural dialogue generation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1489–1498 (2018)
Peters, M., Ruder, S., Smith, N.A.: To tune or not to tune? adapting pretrained representations to diverse tasks. arXiv preprint arXiv:1903.05987 (2019)
Peters, M.E., et al.: Deep contextualized word representations. arXiv preprint arXiv:1802.05365 (2018)
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training. https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/languageunsupervised/languageunderstandingpaper.pdf (2018)
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Languagemodels are unsupervised multitask learners. OpenAI Blog 1(8) (2019)
Sun, Y., et al.: Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223 (2019)
Vaswani, A., et al.: Attention is all you need. In: Advances in neural information processing systems, pp. 5998–6008 (2017)
Wolf, T., Sanh, V., Chaumond, J., Delangue, C.: Transfertransfo: a transfer learning approach for neural network based conversational agents. arXiv preprint arXiv:1901.08149 (2019)
Wu, W., et al.: Proactive human-machine conversation with explicit conversation goals. arXiv preprint arXiv:1906.05572 (2019)
Yavuz, S., Rastogi, A., Chao, G.l., Hakkani-Tür, D., AI, A.A.: Deepcopy: grounded response generation with hierarchical pointer networks. In: NIPS (2018)
Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., Weston, J.: Personalizing dialogue agents: i have a dog, do you have pets too? arXiv preprint arXiv:1801.07243 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Luo, A., Su, C., Pan, S. (2019). Goal-Oriented Knowledge-Driven Neural Dialog Generation System. In: Tang, J., Kan, MY., Zhao, D., Li, S., Zan, H. (eds) Natural Language Processing and Chinese Computing. NLPCC 2019. Lecture Notes in Computer Science(), vol 11839. Springer, Cham. https://doi.org/10.1007/978-3-030-32236-6_64
Download citation
DOI: https://doi.org/10.1007/978-3-030-32236-6_64
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-32235-9
Online ISBN: 978-3-030-32236-6
eBook Packages: Computer ScienceComputer Science (R0)
