Abstract
Link prediction refers to using existing facts in the knowledge graph to predict missing facts. This process can enhance the integrity of the knowledge graph and facilitate various downstream applications. However, existing link prediction models usually extract features only in a global or local scope, resulting in feature extraction being limited to a single scope. Additionally, to achieve optimal results, many models require increasing embedding dimensions and parameter numbers, which can lead to scalability issues when applied to large knowledge graphs. To address these issues, we propose a model that fuses the self-attention mechanism with 2D convolution for the link prediction task. The model utilizes a self-attention mechanism with numerous heads to capture feature interactions between entities and relations in the global scope. Furthermore, we innovatively introduce 2D convolution to capture feature interactions in the local scope. Results using FB15k-237 and WN18RR as standard link prediction benchmarks show that our fusion model has good comparable performance with current state-of-the-art models. In particular, compared to the ConvE model (which uses only 2D convolution), our proposed model achieves 13.7% and 14.7% improvement in MRR metrics, and compared to the SAttLE model (which uses only the self-attention mechanism) achieves 2.5% and 0.5% improvement in MRR metrics. Furthermore, due to the low-dimensional embedding of entities and relations, our proposed model has low complexity, good scalability, and thus can accomplish link prediction tasks on larger knowledge graphs in the real world.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Availability of data and materials
The two datasets used in this research work are publicly available and can be downloaded from the website below. (1)FB15k-237(https://github.com/simonepri/datasets-knowledge-embedding). (2)WN18RR(https://github.com/TimDettmers/ConvE).
Code availability
If requested by the journal, we can provide all our own codes.
References
Zhong L, Wu J, Li Q, Peng H, Wu X (2023) A comprehensive survey on automatic knowledge graph construction. ACM Comput Surv 56(4):1–62. https://doi.org/10.1145/3618295
Zhao X, Chen H, Xing Z, Miao C (2023) Brain-inspired search engine assistant based on knowledge graph. IEEE Transactions on Neural Networks and Learning Systems. 34(8):4386–4400. https://doi.org/10.1109/tnnls.2021.3113026
Liu J, Schmid F, Li K, Zheng W (2021) A knowledge graph-based approach for exploring railway operational accidents. Reliability Engineering & System Safety. 207:107352. https://doi.org/10.1016/j.ress.2020.107352
Chen X, Jia S, Xiang Y (2020) A review: Knowledge reasoning over knowledge graph. Expert Syst Appl 141:112948. https://doi.org/10.1016/j.eswa.2019.112948
Bollacker K, Evans C, Paritosh P, Sturge T, Taylor J (2008) Freebase: a collaboratively created graph database for structuring human knowledge. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data. SIGMOD/PODS ’08. https://doi.org/10.1145/1376616.1376746
Vrandečić D, Krötzsch M (2014) Wikidata: a free collaborative knowledgebase. Commun ACM 57(10):78–85. https://doi.org/10.1145/2629489
Wang Q, Ji Y, Hao Y, Cao J (2020) Grl: Knowledge graph completion with gan-based reinforcement learning. Knowl-Based Syst 209:106421. https://doi.org/10.1016/j.knosys.2020.106421
Shen T, Zhang F, Cheng J (2022) A comprehensive overview of knowledge graph completion. Knowl-Based Syst 255:109597. https://doi.org/10.1016/j.knosys.2022.109597
Lu H, Hu H, Lin X (2022) Dense: An enhanced non-commutative representation for knowledge graph embedding with adaptive semantic hierarchy. Neurocomputing 476:115–125. https://doi.org/10.1016/j.neucom.2021.12.079
Sha X, Sun Z, Zhang J (2021) Hierarchical attentive knowledge graph embedding for personalized recommendation. Electron Commer Res Appl 48:101071. https://doi.org/10.1016/j.elerap.2021.101071
Ji S, Pan S, Cambria E, Marttinen P, Yu PS (2022) A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems. 33(2):494–514. https://doi.org/10.1109/tnnls.2021.3070843
Bordes A, Usunier N, Garcia-Duran A, Weston J, Yakhnenko O (2013) Translating embeddings for modeling multi-relational data. Advances in neural information processing systems 26
Sun Z, Deng Z-H, Nie J-Y, Tang J (2019) Rotate: Knowledge graph embedding by relational rotation in complex space. In: International Conference on Learning Representations
Balazevic I, Allen C, Hospedales T (2019) Tucker: Tensor factorization for knowledge graph completion. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). https://doi.org/10.18653/v1/d19-1522
Dettmers T, Minervini P, Stenetorp P, Riedel S (2018) Convolutional 2d knowledge graph embeddings. Proceedings of the AAAI Conference on Artificial Intelligence. 32(1). https://doi.org/10.1609/aaai.v32i1.11573
Jiang X, Wang Q, Wang B (2019) Adaptive convolution for multi-relational learning. In: Proceedings of the 2019 Conference of the North. https://doi.org/10.18653/v1/n19-1103
Zhou J, Cui G, Hu S, Zhang Z, Yang C, Liu Z, Wang L, Li C, Sun M (2020) Graph neural networks: A review of methods and applications. AI Open. 1:57–81. https://doi.org/10.1016/j.aiopen.2021.01.001
Vashishth S, Sanyal S, Nitin V, Talukdar P (2020) Composition-based multi-relational graph convolutional networks. In: International Conference on Learning Representations
Nathani D, Chauhan J, Sharma C, Kaul M (2019) Learning attention-based embeddings for relation prediction in knowledge graphs. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . https://doi.org/10.18653/v1/p19-1466
Wang X, He Q, Liang J, Xiao Y (2022) Language models as knowledge embeddings. In: Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence. IJCAI-2022. https://doi.org/10.24963/ijcai.2022/318
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. Advances in neural information processing systems. 30
Baghershahi P, Hosseini R, Moradi H (2023) Self-attention presents low-dimensional knowledge graph embeddings for link prediction. Knowl-Based Syst 260:110124. https://doi.org/10.1016/j.knosys.2022.110124
Bi Z, Cheng S, Chen J, Liang X, Xiong F, Zhang N (2024) Relphormer: Relational graph transformer for knowledge graph representations. Neurocomputing 566:127044. https://doi.org/10.1016/j.neucom.2023.127044
Chen S, Liu X, Gao J, Jiao J, Zhang R, Ji Y (2021) Hitter: Hierarchical transformers for knowledge graph embeddings. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/2021.emnlp-main.812
Balažević I, Allen C, Hospedales TM (2019) Hypernetwork Knowledge Graph Embeddings, pp. 553–565. https://doi.org/10.1007/978-3-030-30493-5_52
Balažević I, Allen C, Hospedales T (2019) Multi-relational poincaré graph embeddings. Advances in Neural Information Processing Systems 32 (NIPS 2019). 32:4465–4475
Chami I, Wolf A, Juan D-C, Sala F, Ravi S, Ré C (2020) Low-dimensional hyperbolic knowledge graph embeddings. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.617
Vashishth S, Sanyal S, Nitin V, Agrawal N, Talukdar P (2020) Interacte: Improving convolution-based knowledge graph embeddings by increasing feature interactions. Proceedings of the AAAI Conference on Artificial Intelligence. 34(03):3009–3016. https://doi.org/10.1609/aaai.v34i03.5694
Wang L, Zhao W, Wei Z, Liu J (2022) Simkgc: Simple contrastive knowledge graph completion with pre-trained language models. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). https://doi.org/10.18653/v1/2022.acl-long.295
Das R, Dhuliawala S, Zaheer M, Vilnis L, Durugkar I, Krishnamurthy A, Smola AJ, McCallum A (2018) Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. In: International Conference on Learning Representations
Sadeghian A, Armandpour M, Ding P, Wang DZ (2019) Drum: End-to-end differentiable rule mining on knowledge graphs. Adv Neural Inf Process Syst 32:15321–15331
Pan X, Ge C, Lu R, Song S, Chen G, Huang Z, Huang G (2022) On the integration of self-attention and convolution. In: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr52688.2022.00089
Wu L, Li J, Wang Y, Meng Q, Qin T, Chen W, Zhang M, Liu T-Y et al (2021) R-drop: Regularized dropout for neural networks. Adv Neural Inf Process Syst 34:10890–10905
Toutanova K, Chen D, Pantel P, Poon H, Choudhury P, Gamon M (2015) Representing text for joint embedding of text and knowledge bases. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. https://doi.org/10.18653/v1/d15-1174
Rossi A, Barbosa D, Firmani D, Matinata A, Merialdo P (2021) Knowledge graph embedding for link prediction: A comparative analysis. ACM Trans Knowl Discov Data 15(2):1–49. https://doi.org/10.1145/3424672
Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, Desmaison A, Köpf A, Yang E, DeVito Z, Raison M, Tejani A, Chilamkurthy S, Steiner B, Fang L, Bai J, Chintala S (2019) Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems. 32
Funding
This study was funded by the National Natural Science Foundation of China (319701015), the Natural Science Foundation of Heilongjiang Province (LH2021F037), and the Natural Science Foundation of Heilongjiang Province (JJ2024LH1324).
Author information
Authors and Affiliations
Contributions
Work Concept and design: Shijie Zan, Weidong Ji, Guohui Zhou; Model building: Shijie Zan; Datasets: Shijie Zan, Guohui Zhou; Drafting the thesis: Shijie Zan; Important revisions to the paper: Shijie Zan, Weidong Ji; Approve the final version of the paper to be published: Weidong Ji.
Corresponding author
Ethics declarations
Conflict of interest
All authors certify that they have no affiliations with or involvement in any organization or entity with any financial interest or non-financial interest in the subject matter or materials discussed in this manuscript.
Ethics approval
For the use of the link prediction task dataset, both the WN18RR and FB15k-237 dataset are open.
Consent to participate
Informed consent was obtained from all individual participants included in the study.
Consent for publication
The manuscript has not been sent to any other journal for publication.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zan, S., Ji, W. & Zhou, G. Knowledge graph embeddings based on 2d convolution and self-attention mechanisms for link prediction. Appl Intell 55, 104 (2025). https://doi.org/10.1007/s10489-024-05977-y
Accepted:
Published:
DOI: https://doi.org/10.1007/s10489-024-05977-y