Skip to main content
SpringerLink
Log in

Implicit Relation Inference with Deep Path Extraction for Commonsense Question Answering

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Natural language inference plays an essential role in Commonsense Question Answering. Conventional models usually adopt keywords in questions and choices as queries to retrieve static and explicit evidence that is used to obtain final answers, where dynamic interaction between different keywords and implicit relations inference of deeper information are often neglected. In this paper, we propose a novel joint model, the Graph Relation retrieval Reasoning Network (GRRN), to explicitly introduce the dynamic interaction among different keywords and generate informative features that contribute to representation updating. In addition, to pursue in-depth relations between different keywords, we develop an optimised Path Evidence Fusion in the GRRN to obtain evidence based on deep paths and implicit relations with comprehensive knowledge by making full use of the original paths in external knowledge graphs. The experimental results show that compared with the baselines, our approach achieves remarkable improvement of 1.74\(\%\) for precision on the CommonsenseQA dataset, thereby demonstrating the superiority of our state-of-the-art approach on implicit relation inference with deep paths.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. https://dumps.wikimedia.org/enwiki/.

  2. https://www.elastic.co/.

  3. https://dictionary.cambridge.org/.

  4. https://pytorch.org/.

References

  1. Battaglia PW, Hamrick JB, Bapst V, Sanchez-Gonzalez A, Zambaldi V, Malinowski M, Tacchetti A, Raposo D, Santoro A, Faulkner R, et al. (2018) Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261

  2. Bordes A, Usunier N, Garcia-Durán A, Weston J, Yakhnenko O (2013) Translating embeddings for modeling multi-relational data. In: Proceedings of the 26th International Conference on Neural Information Processing Systems, vol 2, pp 2787–2795

  3. Bosselut A, Rashkin H, Sap M, Malaviya C, Celikyilmaz A, Choi Y (2019) Comet: Commonsense transformers for automatic knowledge graph construction. In: Proceedings of the 57th annual meeting of the association for computational linguistics, pp 4762–4779

  4. Bosselut A, Le Bras R, Choi Y (2021) Dynamic neuro-symbolic knowledge graph construction for zero-shot commonsense question answering. Proceedings of the AAAI Conference on Artificial Intelligence 35:4923–4931

    Article  Google Scholar 

  5. Chen D, Fisch A, Weston J, Bordes A (2017) Reading wikipedia to answer open-domain questions. In: Proceedings of the 55th annual meeting of the association for computational linguistics, vol 1, pp 1870–1879

  6. Chen Q, Ji F, Chen H, Zhang Y (2020) Improving commonsense question answering by graph-based iterative retrieval over multiple knowledge sources. arXiv preprint arXiv:2011.02705

  7. Cho K, van Merrienboer B, Gülçehre Ç, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using rnn encoder-decoder for statistical machine translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp 1724–1734

  8. Clark K, Luong MT, Le QV, Manning CD (2020) Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555

  9. Cui W, Xiao Y, Wang H, Song Y, Hwang Sw, Wang W (2019) Kbqa: learning question answering over qa corpora and knowledge bases. arXiv preprint arXiv:1903.02419

  10. Devlin J, Chang MW, Lee K, Toutanova K (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805

  11. Doxolodeo K, Mahendra R (2020) Ui at semeval-2020 task 4: Commonsense validation and explanation by exploiting contradiction. In: Proceedings of the Fourteenth Workshop on Semantic Evaluation, pp 614–619

  12. Fadel A, Al-Ayyoub M, Cambria E (2020) Justers at semeval-2020 task 4: Evaluating transformer models against commonsense validation and explanation. In: Proceedings of the fourteenth workshop on semantic evaluation, pp 535–542

  13. Feng Y, Chen X, Lin BY, Wang P, Yan J, Ren X (2020) Scalable multi-hop relational reasoning for knowledge-aware question answering. arXiv preprint arXiv:2005.00646

  14. Gretz S, Bilu Y, Cohen-Karlik E, Slonim N (2020) The workweek is the best time to start a family–a study of gpt-2 based claim generation. arXiv preprint arXiv:2010.06185

  15. Huang Y, Fang M, Zhan X, Cao Q, Liang X, Lin L (2021) Rem-net: Recursive erasure memory network for commonsense evidence refinement. Proceedings of the AAAI Conference on Artificial Intelligence 35:6375–6383

    Article  Google Scholar 

  16. He X, Liu Q, Yang Y (2020) Mv-gnn: multi-view graph neural network for compression artifacts reduction. IEEE Trans Image Process 29:6829–6840

    Article  MATH  Google Scholar 

  17. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  18. Huang X, Zhang J, Li D, Li P (2019) Knowledge graph embedding based question answering. In: Proceedings of the twelfth ACM international conference on web search and data mining, pp 105–113

  19. Ji S, Pan S, Cambria E, Marttinen P, Yu PS (2022) A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Trans Neural Networks Learn Syst 33(2):494–514

    Article  MathSciNet  Google Scholar 

  20. Joshi M, Chen D, Liu Y, Weld DS, Zettlemoyer L, Levy O (2020) Spanbert: improving pre-training by representing and predicting spans. Trans Assoc Comput Linguist 8:64–77

    Article  Google Scholar 

  21. Khashabi D, Khot T, Sabharwal A, Tafjord O, Clark P, Hajishirzi H (2020) Unifiedqa: Crossing format boundaries with a single qa system. arXiv preprint arXiv:2005.00700

  22. Kipf TN, Welling M (2016) Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907

  23. Lan Z, Chen M, Goodman S, Gimpel K, Sharma P, Soricut R (2019) Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942

  24. Li L, Gan Z, Cheng Y, Liu J (2019a) Relation-aware graph attention network for visual question answering. In: Proceedings of the IEEE/cvf international conference on computer vision, pp 10313–10322

  25. Li S, Chen J, Yu D (2019b) Teaching pretrained models with commonsense reasoning: a preliminary kb-based approach. arXiv preprint arXiv:1909.09743

  26. Lin BY, Chen X, Chen J, Ren X (2019) Kagnet: Knowledge-aware graph networks for commonsense reasoning. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 2822–2832

  27. Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

  28. Liu Y, Wan Y, He L, Peng H, Yu PS (2021) Kg-bart: knowledge graph augmented bart for generative commonsense reasoning. Proceedings of the AAAI Conference on Artificial Intelligence 35:6418–6425

    Article  Google Scholar 

  29. Loshchilov I, Hutter F (2017) Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101

  30. Lukovnikov D, Fischer A, Lehmann J, Auer S (2017) Neural network-based question answering over knowledge graphs on word and character level. In: Proceedings of the 26th international conference on World Wide Web, pp 1211–1220

  31. Lv S, Guo D, Xu J, Tang D, Duan N, Gong M, Shou L, Jiang D, Cao G, Hu S (2020) Graph-based reasoning over heterogeneous external knowledge for commonsense question answering. Proceedings of the AAAI Conference on Artificial Intelligence 34:8449–8456

    Article  Google Scholar 

  32. Ma K, Francis J, Lu Q, Nyberg E, Oltramari A (2019) Towards generalizable neuro-symbolic systems for commonsense question answering. arXiv preprint arXiv:1910.14087

  33. Ma K, Ilievski F, Francis J, Bisk Y, Nyberg E, Oltramari A (2021) Knowledge-driven data construction for zero-shot evaluation in commonsense question answering. Proceedings of the AAAI Conference on Artificial Intelligence 35:13507–13515

    Article  Google Scholar 

  34. Mihaylov T, Clark P, Khot T, Sabharwal A (2018) Can a suit of armor conduct electricity? a new dataset for open book question answering. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp 2381–2391

  35. Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781

  36. Mishra BD, Tandon N, Clark P (2017) Domain-targeted, high precision knowledge extraction. Trans Assoc Comput Linguist 5:233–246

    Article  Google Scholar 

  37. Mitra A, Banerjee P, Pal KK, Mishra S, Baral C (2019) How additional knowledge can improve natural language commonsense question answering? arXiv preprint arXiv:1909.08855

  38. Na SH, Lee JH (2020) Jbnu at semeval-2020 task 4: Bert and unilm for commonsense validation and explanation. In: Proceedings of the fourteenth workshop on semantic evaluation, pp 527–534

  39. Nickel M, Murphy K, Tresp V, Gabrilovich E (2015) A review of relational machine learning for knowledge graphs. Proc IEEE 104(1):11–33

    Article  Google Scholar 

  40. Nickel M, Rosasco L, Poggio T (2016) Holographic embeddings of knowledge graphs. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pp 1955–1961

  41. Peters M, Neumann M, Iyyer M, Gardner M, Clark C, Lee K, Zettlemoyer L (2018) Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol 1, pp 2227–2237

  42. Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, Zhou Y, Li W, Liu PJ (2019) Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683

  43. Romero J, Razniewski S, Pal K, Z Pan J, Sakhadeo A, Weikum G (2019) Commonsense properties from query logs and question answering forums. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, pp 1411–1420

  44. Saeedi S, Panahi A, Saeedi S, Fong AC (2020) Cs-nlp team at semeval-2020 task 4: evaluation of state-of-the-art nlp deep learning architectures on commonsense reasoning task. arXiv preprint arXiv:2006.01205

  45. Saha A, Pahuja V, Khapra MM, Sankaranarayanan K, Chandar S (2018) Complex sequential question answering: towards learning to converse over linked question answer pairs with a knowledge graph. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, pp 705–713

  46. Santoro A, Raposo D, Barrett DG, Malinowski M, Pascanu R, Battaglia P, Lillicrap T (2017) A simple neural network module for relational reasoning. In: Advances in neural information processing systems, pp 4967–4976

  47. Sap M, Le Bras R, Allaway E, Bhagavatula C, Lourie N, Rashkin H, Roof B, Smith NA, Choi Y (2019) Atomic: An atlas of machine commonsense for if-then reasoning. Proceedings of the AAAI Conference on Artificial Intelligence 33:3027–3035

    Article  Google Scholar 

  48. Speer R, Chin J, Havasi C (2017) Conceptnet 5.5: an open multilingual graph of general knowledge. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, pp 4444–4451

  49. Schlichtkrull M, Kipf TN, Bloem P, Van Den Berg R, Titov I, Welling M (2018) Modeling relational data with graph convolutional networks. In: European semantic web conference, Springer, pp 593–607

  50. Speer R, Chin J, Havasi C (2017) Conceptnet 5.5: an open multilingual graph of general knowledge. In: Proceedings of the AAAI conference on artificial intelligence, vol 31

  51. Strathearn C, Gkatzia D (2021) Chefbot: A novel framework for the generation of commonsense-enhanced responses for task-based dialogue systems. In: Proceedings of the 14th International Conference on Natural Language Generation, pp 46–47

  52. Sun H, Dhingra B, Zaheer M, Mazaitis K, Salakhutdinov R, Cohen W (2018) Open domain question answering using early fusion of knowledge bases and text. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp 4231–4242

  53. Talmor A, Herzig J, Lourie N, Berant J (2019) Commonsenseqa: A question answering challenge targeting commonsense knowledge. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, vol 1, pp 4149–4158

  54. Tandon N, De Melo G, Weikum G (2017) Webchild 2.0: fine-grained commonsense knowledge distillation. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics-System Demonstration, pp 115–120

  55. Wan G, Pan S, Gong C, Zhou C, Haffari G (2021) Reasoning like human: Hierar-chical reinforcement learning for knowledge graph reasoning. In: Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp 1926–1932

  56. Wang C, Liang S, Jin Y, Wang Y, Zhu X, Zhang Y (2020a) Semeval-2020 task 4: commonsense validation and explanation. arXiv preprint arXiv:2007.00236

  57. Wang G, Hou X, Yang D, McKeown K, Huang J (2021) Semantic categorization of social knowledge for commonsense question answering. arXiv preprint arXiv:2109.05168

  58. Wang H, Tang X, Lai S, Leung KS, Zhu J, Fung GPC, Wong KF (2020b) Cuhk at semeval-2020 task 4: Commonsense explanation, reasoning and prediction with multi-task learning. arXiv preprint arXiv:2006.09161

  59. Wang P, Peng N, Szekely P, Ren X (2020c) Connecting the dots: a knowledgeable path generator for commonsense question answering. arXiv preprint arXiv:2005.00691

  60. Wang X, Kapanipathi P, Musa R, Yu M, Talamadupula K, Abdelaziz I, Chang M, Fokoue A, Makni B, Mattei N et al (2019) Improving natural language inference using external knowledge in the science questions domain. Proceedings of the AAAI Conference on Artificial Intelligence 33:7208–7215

    Article  Google Scholar 

  61. Wu S, Li Y, Zhang D, Zhou Y, Wu Z (2020) Topicka: generating commonsense knowledge-aware dialogue responses towards the recommended topic fact. In: Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp 3766–3772

  62. Yan J, Raman M, Zhang T, Rossi R, Zhao H, Kim S, Lipka N, Ren X (2020) Learning contextualized knowledge structures for commonsense reasoning. arXiv preprint arXiv:2010.12873

  63. Yang Z, Dai Z, Yang Y, Carbonell J, Salakhutdinov RR, Le QV (2019) Xlnet: generalized autoregressive pretraining for language understanding. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp 5753–5763

  64. Yasunaga M, Ren H, Bosselut A, Liang P, Leskovec J (2021) Qa-gnn: Reasoning with language models and knowledge graphs for question answering. arXiv preprint arXiv:2104.06378

  65. Ye ZX, Chen Q, Wang W, Ling ZH (2019) Align, mask and select: a simple method for incorporating commonsense knowledge into language representation models. arXiv preprint arXiv:1908.06725

  66. Young T, Cambria E, Chaturvedi I, Zhou H, Biswas S, Huang M (2018) Augmenting end-to-end dialogue systems with commonsense knowledge. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, pp 4970–4977

  67. Zhan X, Huang Y, Dong X, Cao Q, Liang X (2022) Pathreasoner: Explainable reasoning paths for commonsense question answering. Knowl Based Syst 235:107612

    Article  Google Scholar 

  68. Zhang Y, Dai H, Kozareva Z, Smola A, Song L (2018) Variational reasoning for question answering with knowledge graph. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, pp 6069–6076

  69. Zhu C, Cheng Y, Gan Z, Sun S, Goldstein T, Liu J (2019) Freelb: Enhanced adversarial training for natural language understanding. arXiv preprint arXiv:1909.11764

  70. Zhan X, Huang Y, Dong X, Cao Q, Liang X (2022) Pathreasoner: Explainable reasoning paths for commonsense question answering. Knowl Based Syst 235:107612

    Article  Google Scholar 

  71. Huang Y, Fang M, Zhan X, Cao Q, Liang X, Lin L (2021) Rem-net: Recursive erasure memory network for commonsense evidence refinement. Proceedings of the AAAI Conference on Artificial Intelligence 35:6375–6383

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the Consulting Project of Chinese Academy of Engineering under Grant 2020-XY-5, 2018-XY-07, and in part by the Fundamental Research Funds for the Central Universities and the Academy-Locality Cooperation Project of Chinese Academy of Engineering under Grant JS2021ZT05.

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Southeast University, Nanjing, China

    Peng Yang, Zijian Liu & Bing Li

  2. School of Cyber Science and Engineering, Southeast University, Nanjing, China

    Peng Yang & Penghui Zhang

  3. Key Laboratory of Computer Network and Information Integration (Southeast University), Ministry of Education, Nanjing, China

    Peng Yang, Zijian Liu, Bing Li & Penghui Zhang

Authors
  1. Peng Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zijian Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bing Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Penghui Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Peng Yang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, P., Liu, Z., Li, B. et al. Implicit Relation Inference with Deep Path Extraction for Commonsense Question Answering. Neural Process Lett 54, 4751–4768 (2022). https://doi.org/10.1007/s11063-022-10831-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-022-10831-8

Keywords

Search

Navigation