Skip to main content

Advertisement

Knowledge graph embeddings based on 2d convolution and self-attention mechanisms for link prediction

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Link prediction refers to using existing facts in the knowledge graph to predict missing facts. This process can enhance the integrity of the knowledge graph and facilitate various downstream applications. However, existing link prediction models usually extract features only in a global or local scope, resulting in feature extraction being limited to a single scope. Additionally, to achieve optimal results, many models require increasing embedding dimensions and parameter numbers, which can lead to scalability issues when applied to large knowledge graphs. To address these issues, we propose a model that fuses the self-attention mechanism with 2D convolution for the link prediction task. The model utilizes a self-attention mechanism with numerous heads to capture feature interactions between entities and relations in the global scope. Furthermore, we innovatively introduce 2D convolution to capture feature interactions in the local scope. Results using FB15k-237 and WN18RR as standard link prediction benchmarks show that our fusion model has good comparable performance with current state-of-the-art models. In particular, compared to the ConvE model (which uses only 2D convolution), our proposed model achieves 13.7% and 14.7% improvement in MRR metrics, and compared to the SAttLE model (which uses only the self-attention mechanism) achieves 2.5% and 0.5% improvement in MRR metrics. Furthermore, due to the low-dimensional embedding of entities and relations, our proposed model has low complexity, good scalability, and thus can accomplish link prediction tasks on larger knowledge graphs in the real world.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Algorithm 1
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Availability of data and materials

The two datasets used in this research work are publicly available and can be downloaded from the website below. (1)FB15k-237(https://github.com/simonepri/datasets-knowledge-embedding). (2)WN18RR(https://github.com/TimDettmers/ConvE).

Code availability

If requested by the journal, we can provide all our own codes.

References

  1. Zhong L, Wu J, Li Q, Peng H, Wu X (2023) A comprehensive survey on automatic knowledge graph construction. ACM Comput Surv 56(4):1–62. https://doi.org/10.1145/3618295

    Article  MATH  Google Scholar 

  2. Zhao X, Chen H, Xing Z, Miao C (2023) Brain-inspired search engine assistant based on knowledge graph. IEEE Transactions on Neural Networks and Learning Systems. 34(8):4386–4400. https://doi.org/10.1109/tnnls.2021.3113026

    Article  Google Scholar 

  3. Liu J, Schmid F, Li K, Zheng W (2021) A knowledge graph-based approach for exploring railway operational accidents. Reliability Engineering & System Safety. 207:107352. https://doi.org/10.1016/j.ress.2020.107352

    Article  MATH  Google Scholar 

  4. Chen X, Jia S, Xiang Y (2020) A review: Knowledge reasoning over knowledge graph. Expert Syst Appl 141:112948. https://doi.org/10.1016/j.eswa.2019.112948

    Article  MATH  Google Scholar 

  5. Bollacker K, Evans C, Paritosh P, Sturge T, Taylor J (2008) Freebase: a collaboratively created graph database for structuring human knowledge. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data. SIGMOD/PODS ’08. https://doi.org/10.1145/1376616.1376746

  6. Vrandečić D, Krötzsch M (2014) Wikidata: a free collaborative knowledgebase. Commun ACM 57(10):78–85. https://doi.org/10.1145/2629489

    Article  MATH  Google Scholar 

  7. Wang Q, Ji Y, Hao Y, Cao J (2020) Grl: Knowledge graph completion with gan-based reinforcement learning. Knowl-Based Syst 209:106421. https://doi.org/10.1016/j.knosys.2020.106421

    Article  MATH  Google Scholar 

  8. Shen T, Zhang F, Cheng J (2022) A comprehensive overview of knowledge graph completion. Knowl-Based Syst 255:109597. https://doi.org/10.1016/j.knosys.2022.109597

    Article  MATH  Google Scholar 

  9. Lu H, Hu H, Lin X (2022) Dense: An enhanced non-commutative representation for knowledge graph embedding with adaptive semantic hierarchy. Neurocomputing 476:115–125. https://doi.org/10.1016/j.neucom.2021.12.079

    Article  MATH  Google Scholar 

  10. Sha X, Sun Z, Zhang J (2021) Hierarchical attentive knowledge graph embedding for personalized recommendation. Electron Commer Res Appl 48:101071. https://doi.org/10.1016/j.elerap.2021.101071

    Article  MATH  Google Scholar 

  11. Ji S, Pan S, Cambria E, Marttinen P, Yu PS (2022) A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems. 33(2):494–514. https://doi.org/10.1109/tnnls.2021.3070843

    Article  MathSciNet  Google Scholar 

  12. Bordes A, Usunier N, Garcia-Duran A, Weston J, Yakhnenko O (2013) Translating embeddings for modeling multi-relational data. Advances in neural information processing systems 26

  13. Sun Z, Deng Z-H, Nie J-Y, Tang J (2019) Rotate: Knowledge graph embedding by relational rotation in complex space. In: International Conference on Learning Representations

  14. Balazevic I, Allen C, Hospedales T (2019) Tucker: Tensor factorization for knowledge graph completion. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). https://doi.org/10.18653/v1/d19-1522

  15. Dettmers T, Minervini P, Stenetorp P, Riedel S (2018) Convolutional 2d knowledge graph embeddings. Proceedings of the AAAI Conference on Artificial Intelligence. 32(1). https://doi.org/10.1609/aaai.v32i1.11573

  16. Jiang X, Wang Q, Wang B (2019) Adaptive convolution for multi-relational learning. In: Proceedings of the 2019 Conference of the North. https://doi.org/10.18653/v1/n19-1103

  17. Zhou J, Cui G, Hu S, Zhang Z, Yang C, Liu Z, Wang L, Li C, Sun M (2020) Graph neural networks: A review of methods and applications. AI Open. 1:57–81. https://doi.org/10.1016/j.aiopen.2021.01.001

    Article  MATH  Google Scholar 

  18. Vashishth S, Sanyal S, Nitin V, Talukdar P (2020) Composition-based multi-relational graph convolutional networks. In: International Conference on Learning Representations

  19. Nathani D, Chauhan J, Sharma C, Kaul M (2019) Learning attention-based embeddings for relation prediction in knowledge graphs. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . https://doi.org/10.18653/v1/p19-1466

  20. Wang X, He Q, Liang J, Xiao Y (2022) Language models as knowledge embeddings. In: Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence. IJCAI-2022. https://doi.org/10.24963/ijcai.2022/318

  21. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. Advances in neural information processing systems. 30

  22. Baghershahi P, Hosseini R, Moradi H (2023) Self-attention presents low-dimensional knowledge graph embeddings for link prediction. Knowl-Based Syst 260:110124. https://doi.org/10.1016/j.knosys.2022.110124

    Article  MATH  Google Scholar 

  23. Bi Z, Cheng S, Chen J, Liang X, Xiong F, Zhang N (2024) Relphormer: Relational graph transformer for knowledge graph representations. Neurocomputing 566:127044. https://doi.org/10.1016/j.neucom.2023.127044

    Article  MATH  Google Scholar 

  24. Chen S, Liu X, Gao J, Jiao J, Zhang R, Ji Y (2021) Hitter: Hierarchical transformers for knowledge graph embeddings. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/2021.emnlp-main.812

  25. Balažević I, Allen C, Hospedales TM (2019) Hypernetwork Knowledge Graph Embeddings, pp. 553–565. https://doi.org/10.1007/978-3-030-30493-5_52

  26. Balažević I, Allen C, Hospedales T (2019) Multi-relational poincaré graph embeddings. Advances in Neural Information Processing Systems 32 (NIPS 2019). 32:4465–4475

  27. Chami I, Wolf A, Juan D-C, Sala F, Ravi S, Ré C (2020) Low-dimensional hyperbolic knowledge graph embeddings. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.617

  28. Vashishth S, Sanyal S, Nitin V, Agrawal N, Talukdar P (2020) Interacte: Improving convolution-based knowledge graph embeddings by increasing feature interactions. Proceedings of the AAAI Conference on Artificial Intelligence. 34(03):3009–3016. https://doi.org/10.1609/aaai.v34i03.5694

    Article  Google Scholar 

  29. Wang L, Zhao W, Wei Z, Liu J (2022) Simkgc: Simple contrastive knowledge graph completion with pre-trained language models. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). https://doi.org/10.18653/v1/2022.acl-long.295

  30. Das R, Dhuliawala S, Zaheer M, Vilnis L, Durugkar I, Krishnamurthy A, Smola AJ, McCallum A (2018) Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. In: International Conference on Learning Representations

  31. Sadeghian A, Armandpour M, Ding P, Wang DZ (2019) Drum: End-to-end differentiable rule mining on knowledge graphs. Adv Neural Inf Process Syst 32:15321–15331

  32. Pan X, Ge C, Lu R, Song S, Chen G, Huang Z, Huang G (2022) On the integration of self-attention and convolution. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr52688.2022.00089

  33. Wu L, Li J, Wang Y, Meng Q, Qin T, Chen W, Zhang M, Liu T-Y et al (2021) R-drop: Regularized dropout for neural networks. Adv Neural Inf Process Syst 34:10890–10905

    MATH  Google Scholar 

  34. Toutanova K, Chen D, Pantel P, Poon H, Choudhury P, Gamon M (2015) Representing text for joint embedding of text and knowledge bases. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/d15-1174

  35. Rossi A, Barbosa D, Firmani D, Matinata A, Merialdo P (2021) Knowledge graph embedding for link prediction: A comparative analysis. ACM Trans Knowl Discov Data 15(2):1–49. https://doi.org/10.1145/3424672

    Article  MATH  Google Scholar 

  36. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Köpf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S (2019) Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems. 32

Download references

Funding

This study was funded by the National Natural Science Foundation of China (319701015), the Natural Science Foundation of Heilongjiang Province (LH2021F037), and the Natural Science Foundation of Heilongjiang Province (JJ2024LH1324).

Author information

Authors and Affiliations

Authors

Contributions

Work Concept and design: Shijie Zan, Weidong Ji, Guohui Zhou; Model building: Shijie Zan; Datasets: Shijie Zan, Guohui Zhou; Drafting the thesis: Shijie Zan; Important revisions to the paper: Shijie Zan, Weidong Ji; Approve the final version of the paper to be published: Weidong Ji.

Corresponding author

Correspondence to Weidong Ji.

Ethics declarations

Conflict of interest

All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript.

Ethics approval

For the use of the link prediction task dataset, both the WN18RR and FB15k-237 dataset are open.

Consent to participate

Informed consent was obtained from all individual participants included in the study.

Consent for publication

The manuscript has not been sent to any other journal for publication.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zan, S., Ji, W. & Zhou, G. Knowledge graph embeddings based on 2d convolution and self-attention mechanisms for link prediction. Appl Intell 55, 104 (2025). https://doi.org/10.1007/s10489-024-05977-y

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10489-024-05977-y

Keywords