Skip to main content
SpringerLink
Log in

Reason more like human: Incorporating meta information into hierarchical reinforcement learning for knowledge graph reasoning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Nowadays, reasoning over knowledge graphs (KGs) has been widely adapted to empower retrieval systems, recommender systems, and question answering systems, generating a surge in research interest. However, recently developed reasoning methods usually lack interpretability, and can hardly tackle the large-scale action space problem over KGs. Inspired by the ability of human hierarchical decision making, we propose a multi-hop reasoning framework with deep reinforcement learning (RL) to fill this gap, which incorporates meta information into hierarchical reasoning over KGs. We first use optimization-based meta learning method to initialize parameters for RL agents, allowing for efficient adaptation for tasks in a few gradient steps. Then, a hierarchical RL framework is designed to decompose reasoning tasks into several sub-tasks and solve them separately, performed more efficient and natural than directly solving the entire problem. We further evaluated our model through different tasks on five real world datasets. The experimental results indicate that our method outperforms state-of-the-art baseline models without losing interpretability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Algorithm 2
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Fang Y, Wang H, Zhao L, Yu F, Wang C (2020) Dynamic knowledge graph based fake-review detection. Appl Intell 50(12):4281–4295

    Article  Google Scholar 

  2. Li Z, Liu H, Zhang Z, Liu T, Xiong NN (2021) Learning knowledge graph embedding with heterogeneous relation attention networks. IEEE Transactions on Neural Networks and Learning Systems PP(99)

  3. Zhang Z, Li Z, Liu H, Xiong NN (2020) Multi-scale dynamic convolutional network for knowledge graph embedding. IEEE Trans Knowl Data Eng PP(99):1–1

    Google Scholar 

  4. Wang S, Du Z, Ding M, Rodríguez-patón A, Song T (2022) KG-DTI: a knowledge graph based deep learning method for drug-target interaction predictions and alzheimer’s disease drug repositions. Appl Intell 52(1):846–857

    Article  Google Scholar 

  5. Bollacker KD, Evans C, Paritosh PK, Sturge T, Taylor J (2008) Freebase: a collaboratively created graph database for structuring human knowledge. In: Proceedings of the ACM SIGMOD international conference on management of data, SIGMOD 2008, June 10-12, 2008. ACM, pp 1247–1250

  6. Vrandečić D, Krötzsch M (2014) Wikidata: a free collaborative knowledgebase. Commun ACM 57(10):78–85

    Article  Google Scholar 

  7. Liu H, Zheng C, Li D, Shen X, Lin K, Wang J, Zhang Z, Zhang Z, Xiong NN (2022) EDMF: Efficient deep matrix factorization with review feature learning for industrial recommender system. IEEE Trans Ind Inform 18(7):4361–4371

    Article  Google Scholar 

  8. Yi X, Mingjing L, Junyong L, Xiaohui C, Gang Z (2022) Iterative rule-guided reasoning over sparse knowledge graphs with deep reinforcement learning. Information Processing & Management 59(5):103040. https://doi.org/10.1016/j.ipm.2022.103040, https://www.sciencedirect.com/science/article/pii/S0306457322001492

    Article  Google Scholar 

  9. Bordes A, Usunier N, Garcia-Durán A, Weston J, Yakhnenko O (2013) Translating embeddings for modeling multi-relational data. In: Proceedings of the 26th International Conference on Neural Information Processing Systems, NIPS 2013 - vol 2. Curran Associates Inc, pp 2787–2795

  10. Sun Z, Deng Z, Nie J, Tang J (2019) Rotate: Knowledge graph embedding by relational rotation in complex space. In: 7Th international conference on learning representations, ICLR 2019, may 6-9. Openreview.net, 2019

  11. Vashishth S, Sanyal S, Nitin V, Agrawal N, Talukdar PP (2020) Interacte: Improving convolution-based knowledge graph embeddings by increasing feature interactions. In: The thirty-fourth AAAI conference on artificial intelligence, AAAI 2020, february 7-12, 2020. AAAI Press, pp 3009–3016

  12. Feng J, Wei Q, Cui J, Chen J (2022) Novel translation knowledge graph completion model based on 2d convolution. Appl Intell 52(3):3266–3275

    Article  Google Scholar 

  13. Galárraga LA, Teflioudi C, Hose K, Suchanek FM (2013) AMIE: Association rule mining under incomplete evidence in ontological knowledge bases. In: The 22nd international world wide web conference, WWW 2013, may 13-17, 2013. International world wide web conferences steering committee / ACM, pp 413– 422

  14. Lei D, Jiang G, Gu X, Sun K, Mao Y, Ren X (2020) Learning collaborative agents with rule guidance for knowledge graph reasoning. In: Proceedings of the 2020 conference on empirical methods in natural language processing, EMNLP 2020, November 16-20, 2020. Assoc Comput Linguist, pp 8541–8547

  15. Meilicke C, Chekol MW, Fink M, Stuckenschmidt H (2020) Reinforced anytime bottom up rule learning for knowledge graph completion. arXiv:2004.04412

  16. Wang H, Jiang S, Yu Z (2020) Modeling of complex internal logic for knowledge base completion. Appl Intell 50(10):3336–3349

    Article  Google Scholar 

  17. Hildebrandt M, Serna JAQ, Ma Y, Ringsquandl M, Joblin M, Tresp V (2020) Reasoning on knowledge graphs with debate dynamics. In: The 34th AAAI conference on artificial intelligence, AAAI 2020, february 7-12, 2020. AAAI Press, pp 4123– 4131

  18. Wan G, Du B (2021) Gaussianpath: a bayesian multi-hop reasoning framework for knowledge graph reasoning. In: Thirty-fifth AAAI conference on artificial intelligence, AAAI 2021, february 2-9, 2021. AAAI Press, pp 4393–4401

  19. Das R, Dhuliawala S, Zaheer M, Vilnis L, Durugkar I, Krishnamurthy A, Smola A, McCallum A (2018) Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. In: The 6th international conference on learning representations, ICLR 2018, april 30 - may 3, 2018, conference track proceedings. Openreview.net

  20. Lv X, Han X, Hou L, Li J, Liu Z, Zhang W, Zhang Y, Kong H, Wu S (2020) Dynamic anticipation and completion for multi-hop reasoning over sparse knowledge graph. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, November 16-20, 2020. Association for Computational Linguistics, pp 5694–5703

  21. Hou Z, Jin X, Li Z, Bai L (2021) Rule-aware reinforcement learning for knowledge graph reasoning. In: Findings of the association for computational linguistics: ACL/IJCNLP 2021, august 1-6, 2021. Association for computational linguistics, pp 4687–4692

  22. Chen L, Cui J, Tang X, Qian Y, Li Y, Zhang Y (2022) Rlpath: a knowledge graph link prediction method using reinforcement learning based attentive relation path searching and representation learning. Appl Intell 52(4):4715–4726

    Article  Google Scholar 

  23. Wan G, Pan S, Gong C, Zhou C, Haffari G (2020) Reasoning like human: Hierarchical reinforcement learning for knowledge graph reasoning. In: Proceedings of the twenty-ninth international joint conference on artificial intelligence, IJCAI 2020, Online, pp 1926–1932

  24. Wang Q, Mao Z, Wang B, Guo L (2017) Knowledge graph embedding: a survey of approaches and applications. IEEE Trans Knowl Data Eng 29(12):2724–2743

    Article  Google Scholar 

  25. Nickel M, Tresp V, Kriegel H (2011) A three-way model for collective learning on multi-relational data. In: Proceedings of the 28th international conference on machine learning, ICML 2011, June 28 - July 2, 2011. Omnipress, pp 809–816

  26. Dettmers T, Minervini P, Stenetorp P, Riedel S (2018) Convolutional 2d knowledge graph embeddings. In: Proceedings of the 32th AAAI Conference on Artificial Intelligence, AAAI 2018, February 2-7, 2018. AAAI Press, pp 1811–1818

  27. Omran PG, Wang K, Wang Z (2018) Scalable rule learning via learning representation. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pp 2149– 2155

  28. Yang F, Yang Z, Cohen WW (2017) Differentiable learning of logical rules for knowledge base reasoning. In: Advances in neural information processing systems 30: Annual conference on neural information processing systems 2017, december 4-9, 2017, Long Beach, CA, USA, pp 2319–2328

  29. Xiong W, Hoang T, Wang WY (2017) Deeppath: a reinforcement learning method for knowledge graph reasoning. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, September 9-11, 2017. Association for Computational Linguistics, pp 564–573

  30. Fu C, Chen T, Qu M, Jin W, Ren X (2019) Collaborative policy learning for open knowledge graph reasoning. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, EMNLP 2019, November 3-7, 2019. Association for Computational Linguistics, pp 2672–2681

  31. Lin XV, Socher R, Xiong C (2018) Multi-hop knowledge graph reasoning with reward shaping. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018, October 31 - November 4, 2018. Association for Computational Linguistics, pp 3243–3253

  32. Niu G, Zhang Y, Li B, Cui P, Liu S, Li J, Zhang X (2020) Rule-Guided Compositional representation learning on knowledge graphs. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 34, pp 2950–2958

  33. Carlson A, Betteridge J, Kisiel B, Settles B Jr, ERH, Mitchell TM (2010) Toward an architecture for never-ending language learning. In: Proceedings of the 24th AAAI conference on artificial intelligence, AAAI 2010, July 11-15, 2010. AAAI Press

  34. Maas AL, Hannun AY, Ng AY, et al. (2013) Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of the 30th international conference on machine learning, ICML 2013, vol 30. Citeseer, p 3

  35. Finn C, Abbeel P, Levine S (2017) Model-agnostic meta-learning for fast adaptation of deep networks. In: Proceedings of the 34th International Conference on Machine Learning, ICML 2017, 6-11 August 2017. Proceedings of machine learning research, vol 70. PMLR, pp 1126–1135

  36. Williams RJ (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach Learn 8(3-4):229–256

    Article  MATH  Google Scholar 

  37. Toutanova K, Chen D, Pantel P, Poon H, Choudhury P, Gamon M (2015) Representing text for joint embedding of text and knowledge bases. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, September 17-21, 2015. The Association for Computational Linguistics, pp 1499–1509

  38. Trouillon T, Welbl J, Riedel S, Gaussier É, Bouchard G (2016) Complex embeddings for simple link prediction. In: Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, June 19-24, 2016. JMLR Workshop and Conference Proceedings, vol 48. New York, NY, USA pp 2071–2080

  39. Kingma DP, Ba J (2015) Adam: a method for stochastic optimization. In: The 3rd international conference on learning representations, ICLR 2015, may 7-9 2015. Conference Track Proceedings, San Diego, CA, USA

Download references

Acknowledgements

The authors would like to express appreciation for the financial support provided by the National Natural Science Foundation of China (41801313) and the Science and technology project of Henan Province (222102210081, 222300420590).

Author information

Authors and Affiliations

  1. Information Engineering University, Zhengzhou, Henan, China

    Yi Xia, Junyong Luo, Mingjing Lan, Gang Zhou, Zhibo Li & Shuo Liu

Authors
  1. Yi Xia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junyong Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mingjing Lan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gang Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhibo Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shuo Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junyong Luo.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xia, Y., Luo, J., Lan, M. et al. Reason more like human: Incorporating meta information into hierarchical reinforcement learning for knowledge graph reasoning. Appl Intell 53, 13293–13308 (2023). https://doi.org/10.1007/s10489-022-04147-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04147-2

Keywords

Search

Navigation