Abstract
In the domain of multi-turn knowledge-grounded dialogues, the sequential coherence among knowledge elements chosen across various conversational turns presents potential cues for knowledge selection. However, this aspect has been largely overlooked in preceding studies. To tackle this issue, the present study introduces an innovative methodology that employs reinforcement learning to enhance knowledge selection in open-domain dialogue systems. By recasting the knowledge selection challenge as a sequential decision-making task and implementing reinforcement learning, the dialogue system is capable of discerning which knowledge to choose based on the conversational context and preceding dialogue turns, thereby generating high-quality responses. The system acquires a reward signal contingent upon the quality of the generated responses and subsequently updates its policy to maximize the expected reward over time. Harnessing the capabilities of reinforcement learning, our proposed method effectively learns to identify the most pertinent knowledge, thereby generating superior-quality responses. The study assesses the proposed approach using multiple open-domain dialogue datasets, demonstrating that it surpasses the performance of prior methodologies.
The research work is supported by National Key R &D Program of China (No.2022YFB3904700), Key Research and Development Program of in Shandong Province (2019JZZY020102), Key Research and Development Program of Jiangsu Province (No.BE2018084), Industrial Internet Innovation and Development Project in 2021 (TC210A02M, TC210804D), Opening Project of Beijing Key Laboratory of Mobile Computing and Pervasive Device.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Dinan, E., Roller, S., Shuster, K., Fan, A., Auli, M., Weston, J.: Wizard of wikipedia: Knowledge-powered conversational agents. In: International Conference on Learning Representations
Fan, L., et al.: MineDojo: building open-ended embodied agents with internet-scale knowledge (2022). https://doi.org/10.48550/ARXIV.2206.08853, https://arxiv.org/abs/2206.08853
Feng, R., Chen, M.: Multi-sensor data fusion for short-term traffic flow prediction: a novel multi-channel data structure integrated with mixed-pointwise convolution and channel attention mechanism. In: Artificial Neural Networks and Machine Learning-ICANN 2022: 31st International Conference on Artificial Neural Networks, Bristol, UK, September 6–9, 2022, Proceedings; Part IV, pp. 731–742. Springer (2022). https://doi.org/10.1007/978-3-031-15937-4_61
Fu, D., Zhang, C., Yu, J., Sun, Q., Zhan, Z.: Improving dialogue generation with commonsense knowledge fusion and selection. In: Knowledge Science, Engineering and Management: 15th International Conference, KSEM 2022, Singapore, August 6–8, 2022, Proceedings, Part I, pp. 93–108. Springer (2022). https://doi.org/10.1007/978-3-031-10983-6_8
Holtzman, A., Buys, J., Forbes, M., Choi, Y.: The curious case of neural text degeneration. In: International Conference on Learning Representations (ICLR) (2020)
Kim, B., Ahn, J., Kim, G.: Sequential latent knowledge selection for knowledge-grounded dialogue. arXiv preprint arXiv:2002.07510 (2020)
Li, Z., Niu, C., Meng, F., Feng, Y., Li, Q., Zhou, J.: Incremental transformer with deliberation decoder for document grounded conversations. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 12–21 (2019)
Lian, R., Xie, M., Wang, F., Peng, J., Wu, H.: Learning to select knowledge for response generation in dialog systems. In: IJCAI International Joint Conference on Artificial Intelligence, p. 5081 (2019)
Ma, Z., Ye, J., Yang, X., Liu, J.: HCLD: a hierarchical framework for zero-shot cross-lingual dialogue system. In: Proceedings of the 29th International Conference on Computational Linguistics, pp. 4492–4498 (2022)
Mirchandani, S., Karamcheti, S., Sadigh, D.: ELLA: exploration through learned language abstraction. arXiv preprint arXiv:2103.05825 (2021)
Mu, J., et al.: Improving intrinsic exploration with language abstractions (2022). https://doi.org/10.48550/ARXIV.2202.08938, https://arxiv.org/abs/2202.08938
Radford, A., et al.: Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 (2021)
Tam, A.C., et al.: Semantic exploration from language abstractions and pretrained representations (2022). https://doi.org/10.48550/ARXIV.2204.05080, https://arxiv.org/abs/2204.05080
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems. vol. 30 (2017)
Wang, C., Li, Y., Fei, C., Huang, X.: Labeled knowledge-based decision making with assumption-based argumentation. In: Knowledge Science, Engineering and Management: 15th International Conference, KSEM 2022, Singapore, August 6–8, 2022, Proceedings, Part I, pp. 450–465. Springer (2022). https://doi.org/10.1007/978-3-031-10983-6_35
Wang, Y., Zhu, X., Zhang, H.: Relation prediction based on source-entity behavior preference modeling via heterogeneous graph pooling. In: Knowledge Science, Engineering and Management: 15th International Conference, KSEM 2022, Singapore, August 6–8, 2022, Proceedings, Part I, pp. 425–436. Springer (2022). https://doi.org/10.1007/978-3-031-10983-6_33
Xu, J., Wang, H., Niu, Z.Y., Wu, H., Che, W.: Knowledge graph grounded goal planning for open-domain conversation generation. In: AAAI Conference on Artificial Intelligence (AAAI) (2020)
Zhang, T., Huang, M., Zhao, L.: Learning structured representation for text classification via reinforcement learning. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 32 (2018)
Zou, P., Teng, Y., Niu, T.: Multi-scale feature extraction and fusion for online knowledge distillation. In: Artificial Neural Networks and Machine Learning-ICANN 2022: 31st International Conference on Artificial Neural Networks, Bristol, UK, September 6–9, 2022, Proceedings; Part IV, pp. 126–138. Springer (2022). https://doi.org/10.1007/978-3-031-15937-4_11
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ma, Z., Ye, J., Cheng, S. (2023). KSRL: Knowledge Selection Based Reinforcement Learning for Knowledge-Grounded Dialogue. In: Jin, Z., Jiang, Y., Buchmann, R.A., Bi, Y., Ghiran, AM., Ma, W. (eds) Knowledge Science, Engineering and Management. KSEM 2023. Lecture Notes in Computer Science(), vol 14120. Springer, Cham. https://doi.org/10.1007/978-3-031-40292-0_16
Download citation
DOI: https://doi.org/10.1007/978-3-031-40292-0_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-40291-3
Online ISBN: 978-3-031-40292-0
eBook Packages: Computer ScienceComputer Science (R0)