Skip to main content
SpringerLink
Log in

Reinforcement learning from constraints and focal entity shifting in conversational KGQA

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

The actual needs of users for information are often hidden in multiple question answering (QA) on the same topic. In order to generate answers to users’ current questions, a conversational QA system relies on getting external information from a knowledge graph (KG) and combining it with conversation history. Therefore, how to make full use of the information of KG and combining it with the conversation environment is the top priority. In a conversational knowledge graph question answering (KGQA) scenario, the follow-up questions are often incomplete and may contain a shift of focal entity. Considering and processing the constrained information which plays the key role in solving complex conversational KGQA is very important. In this paper, we propose a reinforcement learning (RL) model, which uses a dynamically maintained context entity set to capture the shift of the focal entity in the process of conversation. We then use the bidirectional encoder representations from transformers (BERT) pre-training model to obtain the semantic information from context questions and KG paths. Our model learns from not only the 1-hop path but also 2-hop path constraint of the KG at the same time and gives reward rules based on precision and certain rules, respectively. Compared with state-of-the-art methods on ConvQuestions, our model improves mean reciprocal rank (MRR) and precision at 1 (P@1) by a margin of 4.7\(\%\) and 4.3\(\%\), respectively. Experimental results on two datasets demonstrate the effectiveness of our proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

All data analyzed during this study are from the online demo website of CONVEX [5] (https://convex.mpi-inf.mpg.de) and CONQUER [15] (https://conquer.mpi-inf.mpg.de/).

Notes

  1. https://github.com/PhilippChr/wikidata-core-for-QA.

  2. https://convex.mpi-inf.mpg.de.

References

  1. Abujabal A, Roy RS, Yahya M et al (2017a) Quint: interpretable question answering over knowledge bases. In: Proceedings of the 2017 conference on empirical methods in natural language processing (EMNLP), pp 61–66 https://doi.org/10.18653/v1/D17-2011

  2. Abujabal A, Yahya M, Riedewald M et al (2017b) Automated template generation for question answering over knowledge graphs. In: Proceedings of the 26th international conference on world wide web, pp 1191–1200 https://doi.org/10.1145/3038912.3052583

  3. Bast H, Haussmann E (2015) More accurate question answering on freebase. In: Proceedings of the 24th ACM international on conference on information and knowledge management (CIKM), pp 1431–1440. https://doi.org/10.1145/2806416.2806472

  4. Berant J, Chou A, Frostig R et al (2013) Semantic parsing on freebase from question–answer pairs. In: Proceedings of the 2013 conference on empirical methods in natural language processing (EMNLP), pp 1533–1544. http://www.aclweb.org/anthology/P15-1128

  5. Christmann P, Saha Roy R, Abujabal A et al (2019) Look before you hop: conversational question answering over knowledge graphs using judicious context expansion. In: Proceedings of the 28th ACM international conference on information and knowledge management, pp 729–738. https://doi.org/10.1145/3357384.3358016

  6. Das R, Dhuliawala S, Zaheer M et al (2018) Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. In: Proceedings of the international conference on learning representations (ICLR), pp 1–18. https://doi.org/10.48550/arXiv.1711.05851

  7. Devlin J, Chang MW, Lee K et al (2019) Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics (NAACL), pp 4171–4186. https://doi.org/10.18653/v1/N19-1423

  8. Fang H, Tu Y, Wang H et al (2022) Fuzzy-based adaptive optimization of unknown discrete-time nonlinear Markov jump systems with off-policy reinforcement learning. IEEE Trans Fuzzy Syst 30(12):5276–5290. https://doi.org/10.1109/TFUZZ.2022.3171844

    Article  Google Scholar 

  9. Fang H, Zhang M, He S et al (2022) Solving the zero-sum control problem for tidal turbine system: an online reinforcement learning approach. IEEE Trans Cybern. https://doi.org/10.1109/TCYB.2022.3186886

    Article  Google Scholar 

  10. Ferragina P, Scaiella U (2010) Tagme: on-the-fly annotation of short text fragments (by wikipedia entities). In: Proceedings of the 19th ACM international conference on Information and knowledge management (CIKM), pp 1625–1628. https://doi.org/10.1145/1871437.1871689

  11. Guo D, Tang D, Duan N et al (2018) Dialog-to-action: conversational question answering over a large-scale knowledge base. In: Advances in neural information processing systems, vol 31. Curran Associates, Inc., pp 2946–2955. https://proceedings.neurips.cc/paper/2018/file/d63fbf8c3173730f82b150c5ef38b8ff-Paper.pdf

  12. He G, Lan Y, Jiang J et al (2021) Improving multi-hop knowledge base question answering by learning intermediate supervision signals. In: Proceedings of the 14th ACM international conference on web search and data mining, pp 553–561. https://doi.org/10.1145/3437963.3441753

  13. Jia Z, Pramanik S, Saha Roy R et al (2021) Complex temporal question answering on knowledge graphs. In: Proceedings of the 30th ACM international conference on information and knowledge management (CIKM), pp 792–802. https://doi.org/10.1145/3459637.3482416

  14. Kacupaj E, Plepi J, Singh K et al (2021) Conversational question answering over knowledge graphs with transformer and graph attention networks. In: Proceedings of the 16th conference of the European chapter of the association for computational linguistics (EACL), pp 850–862. https://doi.org/10.18653/v1/2021.eacl-main.72

  15. Kaiser M, Saha Roy R, Weikum G (2021) Reinforcement learning from reformulations in conversational question answering over knowledge graphs. In: Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pp 459–469. https://doi.org/10.1145/3404835.3462859

  16. Kingma DP, Ba J (2015) ADAM: a method for stochastic optimization. In: Proceedings of the international conference on learning representations (ICLR) https://doi.org/10.48550/arXiv.1412.6980

  17. Kumar V, Joshi S (2017) Incomplete follow-up question resolution using retrieval based sequence to sequence learning. In: Proceedings of the 40th international acm sigir conference on research and development in information retrieval, pp 705–714. https://doi.org/10.1145/3077136.3080801

  18. Lan Y, Jiang J (2020) Query graph generation for answering multi-hop complex questions from knowledge bases. In: Association for computational linguistics. https://doi.org/10.18653/v1/2020.acl-main.91

  19. Lan Y, Jiang J (2021) Modeling transitions of focal entities for conversational knowledge base question answering. In: Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing, pp 3288–3297. https://doi.org/10.18653/v1/2021.acl-long.255

  20. Lan Y, He G, Jiang J et al (2021) A survey on complex knowledge base question answering: methods, challenges and solutions. In: Proceedings of the thirtieth international joint conference on artificial intelligence (IJCAI), pp 4483–4491. https://doi.org/10.24963/ijcai.2021/611

  21. Li J, Xiong D (2022) KAFSP: knowledge-aware fuzzy semantic parsing for conversational question answering over a large-scale knowledge base. In: Proceedings of the 60th annual meeting of the association for computational linguistics, pp 461–473. https://doi.org/10.18653/v1/2022.acl-long.35

  22. Lin XV, Xiong C, Socher R (2018) Multi-hop knowledge graph reasoning with reward shaping. In: Proceedings of the 2018 conference on empirical methods in natural language processing (EMNLP), pp 3243–3253. https://doi.org/10.18653/v1/D18-1362

  23. Luo K, Lin F, Luo X et al (2018) Knowledge base question answering via encoding of complex query graphs. In: Proceedings of the 2018 conference on empirical methods in natural language processing (EMNLP), pp 2185–2194. https://doi.org/10.18653/v1/D18-1242

  24. Maheshwari G, Trivedi P, Lukovnikov D et al (2019) Learning to rank query graphs for complex question answering over knowledge graphs. In: International semantic web conference, Springer, pp 487–504. https://doi.org/10.1007/978-3-030-30793-6_28

  25. Marion P, Nowak PK, Piccinno F (2021) Structured context and high-coverage grammar for conversational question answering over knowledge graphs. In: Proceedings of the 2021 conference on empirical methods in natural language processing (EMNLP), pp 8813–8829. https://doi.org/10.18653/v1/2021.emnlp-main.695

  26. Plepi J, Kacupaj E, Singh K et al (2021) Context transformer with stacked pointer networks for conversational question answering over knowledge graphs. In: The semantic web: 18th international conference, ESWC 2021, virtual event, June 6–10, 2021, Proceedings, Springer, pp 356–371. https://doi.org/10.1007/978-3-030-77385-4_21

  27. Qiu Y, Zhang K, Wang Y et al (2020) Hierarchical query graph generation for complex question answering over knowledge graph. In: Proceedings of the 29th ACM international conference on information & knowledge management (CIKM), pp 1285–1294. https://doi.org/10.1145/3340531.3411888

  28. Qu C, Yang L, Qiu M et al (2019) Bert with history answer embedding for conversational question answering. In: Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, pp 1133–1136. https://doi.org/10.1145/3331184.3331341

  29. Ren G, Ni X, Malik M et al (2018) Conversational query understanding using sequence to sequence modeling. In: Proceedings of the 2018 World Wide Web Conference, pp 1715–1724. https://doi.org/10.1145/3178876.3186083

  30. Saha A, Pahuja V, Khapra M et al (2018) Complex sequential question answering: towards learning to converse over linked question answer pairs with a knowledge graph. In: Proceedings of the AAAI conference on artificial intelligence. https://doi.org/10.1609/aaai.v32i1.11332

  31. Shi J, Cao S, Hou L et al (2021) Transfernet: an effective and transparent framework for multi-hop question answering over relation graph. In: Proceedings of the 2021 conference on empirical methods in natural language processing (EMNLP), pp 4149–4158. https://doi.org/10.18653/v1/2021.emnlp-main.341

  32. Vakulenko S, Longpre S, Tu Z et al (2021) Question rewriting for conversational question answering. In: Proceedings of the 14th ACM international conference on web search and data mining (WSDM), pp 355–363 https://doi.org/10.1145/3437963.3441748

  33. Vrandečić D, Krötzsch M (2014) Wikidata: a free collaborative knowledgebase. Commun ACM 57(10):78–85. https://doi.org/10.1145/2629489

    Article  Google Scholar 

  34. Xiong W, Hoang T, Wang WY (2017) Deeppath: a reinforcement learning method for knowledge graph reasoning. In: Proceedings of the 2017 conference on Empirical methods in natural language processing (EMNLP), pp 564–573 https://doi.org/10.18653/v1/D17-1060

  35. Yih Wt, Chang MW, He X et al (2015) Semantic parsing via staged query graph generation: Question answering with knowledge base. In: Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing. Association for computational linguistics, pp 1321–1331 http://www.aclweb.org/anthology/P15-1128

  36. Zaib M, Zhang WE, Sheng QZ et al (2022) Conversational question answering: a survey. Knowl Inf Syst 64(12):3151–3195. https://doi.org/10.1007/s10115-022-01744-y

    Article  Google Scholar 

  37. Zhang Q, Weng X, Zhou G et al (2022) ARL: an adaptive reinforcement learning framework for complex question answering over knowledge base. Inf Process Manag 59(3):102933. https://doi.org/10.1016/j.ipm.2022.102933

    Article  Google Scholar 

Download references

Acknowledgements

The research work was supported by the Natural Science Foundation of China under Grant No.U21A20491, No.U1936109, No.U1908214, No.KLSA201906.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Dalian University of Technology, No. 2 Linggong Road, Dalian City, 116024, Liaoning Province, China

    Xirong Xu, Tao Xu, Ziming Wang, Haochen Li & Xiaopeng Wei

  2. School of Control Science and Engineering, Dalian University of Technology, No. 2 Linggong Road, Dalian City, 116024, Liaoning Province, China

    Li Zhu

Authors
  1. Xirong Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tao Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ziming Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Haochen Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Li Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xiaopeng Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xirong Xu.

Ethics declarations

Conflict of interest

The authors declare no conflicts of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, X., Xu, T., Wang, Z. et al. Reinforcement learning from constraints and focal entity shifting in conversational KGQA. Neural Comput & Applic 36, 2015–2028 (2024). https://doi.org/10.1007/s00521-023-09138-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-09138-z

Keywords

Search

Navigation