Skip to main content
SpringerLink
Log in

VEM\(^2\)L: an easy but effective framework for fusing text and structure knowledge on sparse knowledge graph completion

  • Published:
Data Mining and Knowledge Discovery Aims and scope Submit manuscript

Abstract

The task of Knowledge Graph Completion (KGC) is to infer missing links for Knowledge Graphs (KGs) by analyzing graph structures. However, with increasing sparsity in KGs, this task becomes increasingly challenging. In this paper, we propose VEM\(^2\)L, a joint learning framework that incorporates structure and relevant text information to supplement insufficient features for sparse KGs. We begin by training two pre-existing KGC models: one based on structure and the other based on text. Our ultimate goal is to fuse knowledge acquired by these models. To achieve this, we divide knowledge within the models into two non-overlapping parts: expressive power and generalization ability. We then propose two different joint learning methods that co-distill these two kinds of knowledge respectively. For expressive power, we allow each model to learn from and exchange knowledge mutually on training examples. For the generalization ability, we propose a novel co-distillation strategy using the Variational EM algorithm on unobserved queries. Our proposed joint learning framework is supported by both detailed theoretical evidence and qualitative experiments, demonstrating its effectiveness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. Microsoft Research Data License

  2. github.com/TimDettmers/ConvE

References

  • Balazevic I, Allen C, Hospedales T (2019) Tucker: tensor factorization for knowledge graph completion. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). Association for Computational Linguistics

  • Besag J (1975) Statistical analysis of non-lattice data. J R Stat Soc Ser D (The Statistician) 24(3):179–195

    Google Scholar 

  • Bishop CM (2006) Pattern recognition and machine learning, vol 4. Springer, Cham

    Google Scholar 

  • Bordes A, Usunier N, Garcia-Duran A, Weston J, Yakhnenko O (2013) Translating embeddings for modeling multi-relational data. In: Advances in neural information processing systems, vol 26

  • Chen W, Cao Y, Feng F, He X, Zhang Y (2022) Explainable sparse knowledge graph completion via high-order graph reasoning network. arXiv preprint arXiv:2207.07503

  • Chen W, Xiong W, Yan X, Wang WY (2018) Variational knowledge graph reasoning. In: Proceedings of the 2018 conference of the North American chapter of the association for computational linguistics: human language technologies, vol 1. Long Papers, pp 1823–1832

  • Das R, Dhuliawala S, Zaheer M, Vilnis L, Durugkar I, Krishnamurthy A, Smola A, McCallum A (2018) Go for a walk and arrive at the answer: reasoning over paths in knowledge bases using reinforcement learning. In: International conference on learning representations

  • Dettmers T, Minervini P, Stenetorp P, Riedel S (2018) Convolutional 2d knowledge graph embeddings. In: 32nd AAAI conference on artificial intelligence

  • Fu C, Chen T, Qu M, Jin W, Ren X (2019) Collaborative policy learning for open knowledge graph reasoning. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 2672–2681

  • Hinton G, Vinyals O, Dean J et al (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

  • Huang X, Zhang J, Li D, Li P (2019) Knowledge graph embedding based question answering. In: Proceedings of the 12th ACM international conference on web search and data mining, pp 105–113

  • Kenton JDM-WC, Toutanova LK (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT, pp 4171–4186

  • Kipf TN, Welling M (2016) Semi-supervised classification with graph convolutional networks. In: International conference on learning representations

  • Li R, Cao Y, Zhu Q, Bi G, Fang F, Liu Y, Li Q (2022) How does knowledge graph embedding extrapolate to unseen data: a semantic evidence view. In: Proceedings of the AAAI conference on artificial intelligence, vol 36, pp 5781–5791

  • Lin XV, Socher R, Xiong C (2018) Multi-hop knowledge graph reasoning with reward shaping. In: Proceedings of the 2018 conference on empirical methods in natural language processing, pp 3243–3253

  • Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, Levy O, Lewis M, Zettlemoyer L, Stoyanov V (2019) Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692

  • Liu Y, Sun Z, Li G, Hu W (2022) I know what you do not know: knowledge graph embedding via co-distillation learning. In: Proceedings of the 31st ACM international conference on information & knowledge management, pp 1329–1338

  • Lv X, Han X, Hou L, Li J, Liu Z, Zhang W, Zhang Y, Kong H, Wu S (2020) Dynamic anticipation and completion for multi-hop reasoning over sparse knowledge graph. In: Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp 5694–5703

  • Lv X, Lin Y, Cao Y, Hou L, Li J, Liu Z, Li P, Zhou J (2022) Do pre-trained models benefit knowledge graph completion? A reliable evaluation and a reasonable approach. In: Findings of the association for computational linguistics: ACL 2022, pp 3570–3581

  • Malaviya C, Bhagavatula C, Bosselut A, Choi Y (2020) Commonsense knowledge base completion with structural and semantic context. In: Proceedings of the AAAI conference on artificial intelligence, vol 34, pp 2925–2933

  • Markowitz E, Balasubramanian K, Mirtaheri M, Annavaram M, Galstyan A, Ver Steeg G (2022) Statik: structure and text for inductive knowledge graph completion. In: Findings of the association for computational linguistics: NAACL 2022, pp 604–615

  • Nathani D, Chauhan J, Sharma C, Kaul M (2019) Learning attention-based embeddings for relation prediction in knowledge graphs. In: Proceedings of the 57th annual meeting of the association for computational linguistics, pp 4710–4723

  • Neal RM, Hinton GE (1998) A view of the EM algorithm that justifies incremental, sparse, and other variants. Learning in graphical models. Springer, Cham, pp 355–368

    Chapter  Google Scholar 

  • Nickel M, Tresp V, Kriegel H-P (2011) A three-way model for collective learning on multi-relational data. In: ICML

  • Oh B, Seo S, Hwang J, Lee D, Lee K-H (2022) Open-world knowledge graph completion for unseen entities and relations via attentive feature aggregation. Inf Sci 586:468–484

    Article  Google Scholar 

  • Pavlović A, Sallinger E (2022) Expressive: a spatio-functional embedding for knowledge graph completion. In: The 11th international conference on learning representations

  • Qiu J, Chai Y, Tian Z, Du X, Guizani M (2019) Automatic concept extraction based on semantic graphs from big data in smart city. IEEE Trans Comput Soc Syst 7(1):225–233

    Article  Google Scholar 

  • Qu M, Bengio Y, Tang J (2019) Gmnn: graph markov neural networks. In: International conference on machine learning, PMLR, pp 5241–5250

  • Rossi A, Barbosa D, Firmani D, Matinata A, Merialdo P (2021) Knowledge graph embedding for link prediction: A comparative analysis. ACM Trans Knowl Discov Data (TKDD) 15(2):1–49

    Article  Google Scholar 

  • Schlichtkrull M, Kipf TN, Bloem P, Van Den Berg R, Titov I, Welling M (2018) Modeling relational data with graph convolutional networks. In: European semantic web conference, Springer, pp 593–607

  • Shang C, Tang Y, Huang J, Bi J, He X, Zhou B (2019) End-to-end structure-aware convolutional networks for knowledge base completion. In: Proceedings of the AAAI conference on artificial intelligence, vol 33, pp 3060–3067

  • Sun Z, Deng Z-H, Nie J-Y, Tang J (2018) Rotate: knowledge graph embedding by relational rotation in complex space. In: International conference on learning representations

  • Sun Z, Vashishth S, Sanyal S, Talukdar P, Yang Y (2020) A re-evaluation of knowledge graph completion methods. In: Proceedings of the 58th annual meeting of the association for computational linguistics, pp 5516–5522

  • Toutanova K, Chen D, Pantel P, Poon H, Choudhury P, Gamon M (2015) Representing text for joint embedding of text and knowledge bases. In: Proceedings of the 2015 conference on empirical methods in natural language processing, pp 1499–1509

  • Trouillon T, Welbl J, Riedel S, Gaussier É, Bouchard G (2016) Complex embeddings for simple link prediction. In: International conference on machine learning, PMLR, pp 2071–2080

  • Vashishth S, Sanyal S, Nitin V, Agrawal N, Talukdar P (2020) Interacte: improving convolution-based knowledge graph embeddings by increasing feature interactions. In: Proceedings of the AAAI conference on artificial intelligence, vol 34, pp 3009–3016

  • Vashishth S, Sanyal S, Nitin V, Talukdar P (2019) Composition-based multi-relational graph convolutional networks. In: International conference on learning representations

  • Wang K, Liu Y, Ma Q, Sheng QZ (2021) Mulde: multi-teacher knowledge distillation for low-dimensional knowledge graph embeddings. In: Proceedings of the web conference 2021, pp 1716–1726

  • Wang B, Shen T, Long G, Zhou T, Wang Y, Chang Y (2021) Structure-augmented text representation learning for efficient knowledge graph completion. In: Proceedings of the web conference 2021, pp 1737–1748

  • Wang H, Zhang F, Xie X, Guo M (2018) Dkn: deep knowledge-aware network for news recommendation. In: Proceedings of the 2018 world wide web conference, pp 1835–1844

  • Wang L, Zhao W, Wei Z, Liu J (2022) Simkgc: simple contrastive knowledge graph completion with pre-trained language models. arXiv preprint arXiv:2203.02167

  • Xiao C, He X, Cao Y (2023) Knowledge graph embedding by normalizing flows. In: AAAI conference on artificial intelligence

  • Xie R, Liu Z, Jia J, Luan H, Sun M (2016) Representation learning of knowledge graphs with entity descriptions. In: Proceedings of the AAAI conference on artificial intelligence, vol 30

  • Xiong C, Power R, Callan J (2017) Explicit semantic ranking for academic search via knowledge graph embedding. In: Proceedings of the 26th international conference on world wide web, pp 1271–1279

  • Xu J, Qiu X, Chen K, Huang X (2017) Knowledge graph representation with jointly structural and textual encoding. In: Proceedings of the 26th international joint conference on artificial intelligence, pp 1318–1324

  • Yang B, Yih SW-T, He X, Gao J, Deng L (2015) Embedding entities and relations for learning and inference in knowledge bases. In: Proceedings of the international conference on learning representations (ICLR) 2015

  • Yao L, Mao C, Luo Y (2019) Kg-bert: bert for knowledge graph completion. arXiv preprint arXiv:1909.03193

  • Zhang Y, Xiang T, Hospedales TM, Lu H (2018) Deep mutual learning. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4320–4328

  • Zhang Y, Yao Q (2022) Knowledge graph reasoning with relational digraph. In: Proceedings of the ACM web conference 2022, pp 912–924

  • Zhang D, Yuan Z, Liu H, Xiong H et al (2022) Learning to walk with dual agents for knowledge graph reasoning. In: Proceedings of the AAAI Conference on artificial intelligence, vol 36, pp 5932–5941

  • Zhou L, Li J, Gu Z, Qiu J, Gupta BB, Tian Z (2022) Panner: pos-aware nested named entity recognition through heterogeneous graph neural network. IEEE Trans Comput Soc Syst 45:1–9. https://doi.org/10.1109/TCSS.2022.3159366

    Article  Google Scholar 

  • Zhu Z, Zhang Z, Xhonneux L-P, Tang J (2021) Neural bellman-ford networks: a general graph neural network framework for link prediction. Adv Neural Inf Process Syst 34:29476–29490

    Google Scholar 

  • Zhu Y, Zhang W, Chen M, Chen H, Cheng X, Zhang W, Chen H (2022) Dualde: dually distilling knowledge graph embedding for faster and cheaper reasoning. In: Proceedings of the 15th ACM international conference on web search and data mining, pp 1516–1524

Download references

Acknowledgements

The research in this article is supported by the National Key Research and Development Project (2022YFF0903301), the National Science Foundation of China (U22B2059, 61976073, 62276083), and Shenzhen Foundational Research Funding (JCYJ20200109113441941), Major Key Project of PCL (PCL2021A06).

Author information

Authors and Affiliations

  1. Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology, Xidazhi, Harbin, 150001, Heilongjiang, China

    Tao He, Ming Liu, Zihao Zheng & Bing Qin

  2. Peng Cheng Laboratory, Nanshan District, Shenzhen, 518000, Guangdong, China

    Ming Liu & Bing Qin

  3. SMU School of Computing and Information Systems 1, Singapore Management University, 80 Stamford Rd, Singapore, 178902, Singapore

    Yixin Cao

  4. Mila - Quebec AI Institute, 6666 Rue Saint-Urbain, Montreal, QC, H2S 3H1, Canada

    Meng Qu

Authors
  1. Tao He
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ming Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yixin Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Meng Qu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zihao Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Bing Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

TH—Conceptualization, Methodology, Data processing, Programming, Writing; ML—Review & editing; YC—Review & editing; MQ—Review & editing; ZZ—Comparative experiments; BQ—Funding acquisition.

Corresponding author

Correspondence to Ming Liu.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Ethical approval

This paper satisfies the compliance with ethical standards. There is no potential conflicts of interest; The research does not involve Human Participants and/or Animals; The data in this paper has been anonymized to protect data privacy; Informed consent was obtained from all individual participants.

Consent for publication

All individual participants have consented to the submission of the regular paper to the journal.

Additional information

Responsible editor: Tim Weninger.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

He, T., Liu, M., Cao, Y. et al. VEM\(^2\)L: an easy but effective framework for fusing text and structure knowledge on sparse knowledge graph completion. Data Min Knowl Disc 38, 343–371 (2024). https://doi.org/10.1007/s10618-023-01001-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10618-023-01001-y

Keywords

Search

Navigation