Skip to main content
SpringerLink
Log in

New attention strategy for negative sampling in knowledge graph embedding

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In the study of knowledge graph embedding (KGE), self-adversarial negative sampling is a recently proposed technique based on the attention mechanism, which pays more attention to the negative triplets with higher embedding scores. Unfortunately, the technique also pays more attention to those false negative triplets with higher embedding scores, which is obviously unreasonable and often leads to a performance downgrade of the KGE model. To alleviate the downgrade, this paper proposes a new attention strategy, aiming at gradually decreasing the attention of high-score false negative triplets. In the new strategy, the attention difference between high-score and low-score negative triplets will be narrowed as the KGE model performance improves, which is more reasonable during training. Experimental results on TransE, DistMult, RotatE, and PairRE show that our proposed strategy indeed has a significant performance improvement for KGE models on the task of linking prediction and triplet classification.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Algorithm 1
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  1. Wang Q, Mao Z, Wang B, Guo L (2017) Knowledge graph embedding: A survey of approaches and applications. IEEE Trans Knowl Data Eng 29(12):2724–2743. https://doi.org/10.1109/TKDE.2017.2754499

    Article  Google Scholar 

  2. Ji S, Pan S, Cambria E, Marttinen P, Yu PS (2022) A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Trans Neural Netw Learn Syst 33(2):494–514. https://doi.org/10.1109/TNNLS.2021.3070843

    Article  MathSciNet  Google Scholar 

  3. Lenat DB (1995) Cyc: A large-scale investment in knowledge infrastructure. Commun ACM 38:33–38. https://doi.org/10.1145/219717.219745

    Article  Google Scholar 

  4. Bollacker K, Evans C, Paritosh P, Sturge T, Taylor J (2008) Freebase: a collaboratively created graph database for structuring human knowledge. In: Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data. pp. 1247–1250. https://doi.org/10.1145/1376616.1376746

  5. Bizer C, Lehmann J, Kobilarov G, Auer S, Becker C, Cyganiak R, Hellmann S (2009) Dbpedia-a crystallization point for the web of data. J Web Semantics 7(3):154–165. https://doi.org/10.1016/j.websem.2009.07.002

    Article  Google Scholar 

  6. Suchanek FM, Kasneci G, Weikum G (2007) Yago: A core of semantic knowledge. In: Proceedings of the 16th International Conference on World Wide Web. pp. 697–706. https://doi.org/10.1145/1242572.1242667

  7. Miller GA (1995) Wordnet: a lexical database for english. Commun ACM 38(11):39–41. https://doi.org/10.1145/219717.219748

    Article  Google Scholar 

  8. Carlson A, Betteridge J, Kisiel B, Settles B, Hruschka E, Mitchell T (2010) Toward an architecture for never-ending language learning. Proc AAAI Conf Art Intell 24:1306–1313

  9. Wang R , Yan Y, Wang J, Jia Y, Zhang Y, Zhang W, Wang X (2018) Acekg: A large-scale knowledge graph for academic data mining. In: Proceedings of the 27th ACM International Conference on Information and Knowledge Management. pp. 1487–1490. https://doi.org/10.1145/3269206.3269252

  10. Färber M (2019) The microsoft academic knowledge graph: A linked data source with 8 billion triples of scholarly data. In: Proceedings of the 18th International Semantic Web Conference. pp. 113–129. https://doi.org/10.5281/zenodo.2159723

  11. Globerson A, Lazic N, Chakrabarti S, Subramanya A, Ringgaard M, Pereira F (2016) Collective entity resolution with multi-focal attention. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 621–631

  12. Sil A, Florian R (2016) One for all: Towards language independent named entity linking. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 2255–2264

  13. Wang M, Wang R, Liu J, Chen Y, Zhang L, Qi G (2018) Towards empty answers in sparql: approximating querying with rdf embedding. In: International Semantic Web Conference. pp. 513–529. https://doi.org/10.1007/978-3-030-00671-6_30

  14. Wang R, Wang M, Liu J, Yao S, Zheng Q (2018) Graph embedding based query construction over knowledge graphs. In: 2018 IEEE International Conference on Big Knowledge (ICBK). pp. 1–8. https://doi.org/10.1109/ICBK.2018.00009

  15. Dong L, Wei F, Zhou M, Xu K (2015) Question answering over freebase with multi-column convolutional neural networks. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 260–269

  16. Chen Y, Wu L, Zaki MJ (2019) Bidirectional attentive memory networks for question answering over knowledge bases. In: Proceedings of NAACL-HLT. pp. 2913–2923

  17. Ai Q, Azizi V, Chen X, Zhang Y (2018) Learning heterogeneous knowledge base embeddings for explainable recommendation. Algorithms 11(9):137. https://doi.org/10.3390/a11090137

  18. Zhang F, Yuan NJ, Lian D, Xie X, Ma W-Y (2016) Collaborative knowledge base embedding for recommender systems. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 353–362. https://doi.org/10.1145/2939672.2939673

  19. Nickel M, Tresp V, Kriegel H-P (2011) A threeway model for collective learning on multirelational data. In: International Conference on Machine Learning. pp. 809–816

  20. Bordes A, Usunier N, Garcia-Duran A, Weston J, Yakhnenko O (2013) Translating embeddings for modeling multi-relational data. In: Advances in Neural Information Processing Systems, vol. 26

  21. Socher R, Chen D, Manning CD, Ng A (2013) Reasoning with neural tensor networks for knowledge base completion. In: Advances in Neural Information Processing Systems, vol. 26

  22. Wang Z, Zhang J, Feng J, Chen Z (2014) Knowledge graph embedding by translating on hyperplanes. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 28. pp. 1112–1119. https://doi.org/10.1609/aaai.v28i1.8870

  23. Bordes A, Glorot X, Weston J, Bengio Y (2014) A semantic matching energy function for learning with multi-relational data. Mach Learn 94(2):233–259. https://doi.org/10.1007/s10994-013-5363-6

    Article  MathSciNet  MATH  Google Scholar 

  24. Lin Y, Liu Z, Sun M, Liu Y, Zhu X (2015) Learning entity and relation embeddings for knowledge graph completion. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 29. pp. 2181–2187. https://doi.org/10.1609/aaai.v29i1.9491

  25. Ji G, He S, Xu L, Liu K, Zhao J (2015) Knowledge graph embedding via dynamic mapping matrix. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (volume 1: Long Papers). pp. 687–696

  26. Yang B, Yih SW-t, He X, Gao J, Deng L (2015) Embedding entities and relations for learning and inference in knowledge bases. In: International Conference on Learning Representations

  27. He S, Liu K, Ji G, Zhao J (2015) Learning to represent knowledge graphs with gaussian embedding. In: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. pp. 623–632. https://doi.org/10.1145/2806416.2806502

  28. Xiao H, Huang M, Zhu X (2016) Transg: A generativemodel for knowledge graph embedding. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 2316–2325. https://doi.org/10.18653/v1/P16-1219

  29. Nickel M, Rosasco L, Poggio T (2016) Holographic embeddings of knowledge graphs. Proceedings of the AAAI Conference on Artificial Intelligence 30:1955–1961. https://doi.org/10.1609/aaai.v30i1.10314

    Article  Google Scholar 

  30. Trouillon T, Welbl J, Riedel S, Gaussier É, Bouchard G (2016) Complex embeddings for simple link prediction. International Conference on Machine Learning 48:2071–2080

    Google Scholar 

  31. Liu Q, Jiang H, Evdokimov A, Ling Z- H, Zhu X, Wei S, Hu Y (2016) Probabilistic reasoning via deep learning: neural association models. arXiv:1603.07704

  32. Sun Z, Deng Z-H, Nie J-Y, Tang J (2019) Rotate: Knowledge graph embedding by relational rotation in complex space. In: International Conference on Learning Representations

  33. Dai G, Wang X, Zou X, Liu C, Cen S (2022) Mrgat: Multi-relational graph attention network for knowledge graph completion. Neural Networks. https://doi.org/10.1016/j.neunet.2022.07.014

  34. Zhang Y, Yao Q, Shao Y, Chen L (2019) Nscaching: Simple and efficient negative sampling for knowledge graph embedding. In: 2019 IEEE 35th International Conference on Data Engineering (ICDE). pp. 614–625. https://doi.org/10.1109/ICDE.2019.00061

  35. Bayrak B (2020) Effects of negative sampling on knowledge graph completion. In: 2020 5th International Conference on Computer Science and Engineering (UBMK). pp. 264–267. https://doi.org/10.1109/UBMK50275.2020.9219377

  36. Rossi A, Barbosa D, Firmani D, Matinata A, Merialdo P (2021) Knowledge graph embedding for link prediction: A comparative analysis. ACM Trans Knowl Discov Data 15(2). https://doi.org/10.1145/3424672

  37. Chao L, He J, Wang T, Chu W (2021) PairRE: Knowledge graph embeddings via paired relation vectors. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 4360–4369. https://doi.org/10.18653/v1/2021.acl-long.336

  38. Liu H, Wu Y, Yang Y (2017) Analogical inference for multi-relational embeddings. In: Proceedings of the 34th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 70. PMLR. pp. 2168–2178

  39. Kazemi SM, Poole D (2018) Simple embedding for link prediction in knowledge graphs. In: Advances in Neural Information Processing Systems, vol. 31

  40. Dettmers T, Minervini P, Stenetorp P, Riedel S (2018) Convolutional 2d knowledge graph embeddings. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32. pp. 1811–1818. https://doi.org/10.1609/aaai.v32i1.11573

  41. Nguyen DQ, Nguyen TD, Nguyen DQ, Phung D (2018) A novel embedding model for knowledge base completion based on convolutional neural network. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). pp. 327–333. https://doi.org/10.18653/v1/N18-2053

  42. Jiang X, Wang Q, Wang B (2019) Adaptive convolution for multi-relational learning. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 978–987. https://doi.org/10.18653/v1/N19-1103

  43. Vashishth S, Sanyal S, Nitin V, Agrawal N, Talukdar P (2020) Interacte: Improving convolution-based knowledge graph embeddings by increasing feature interactions. Proc AAAI Conf Artif Intell 34(03):3009–3016. https://doi.org/10.1609/aaai.v34i03.5694

    Article  Google Scholar 

  44. Nguyen DQ, Vu T, Nguyen TD, Nguyen DQ, Phung D (2019) A capsule network-based embedding model for knowledge graph completion and search personalization. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 2180–2189. https://doi.org/10.18653/v1/N19-1226

  45. Sabour S, Frosst N, Hinton GE (2017) Dynamic routing between capsules. In: Advances in Neural Information Processing Systems, vol. 30

  46. Schlichtkrull M, Kipf TN, Bloem P, van den Berg R, Titov I, Welling M (2018) Modeling relational data with graph convolutional networks. In: The Semantic Web. pp. 593–607. https://doi.org/10.1007/978-3-319-93417-4_38

  47. Shang C, Tang Y, Huang J, Bi J, He X, Zhou B (2019) End-to-end structure-aware convolutional networks for knowledge base completion. Proc AAAI Conf Artif Intell 33(01):3060–3067. https://doi.org/10.1609/aaai.v33i01.33013060

    Article  Google Scholar 

  48. Nathani D, Chauhan J, Sharma C, Kaul M (2019) Learning attention-based embeddings for relation prediction in knowledge graphs. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 4710–4723. https://doi.org/10.18653/v1/P19-1466

  49. Ahrabian K, Feizi A, Salehi Y, Hamilton WL, Bose AJ (2020) Structure aware negative sampling in knowledge graphs. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pp. 6093–6101. https://doi.org/10.18653/v1/2020.emnlp-main.492

  50. Wang P, Li S, Pan R (2018) Incorporating gan for negative sampling in knowledge representation learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32. pp. 2005–2012. https://doi.org/10.1609/aaai.v32i1.11536

  51. Zhou X, Niu L, Zhu Q, Zhu X, Liu P, Tan J, Guo L (2021) Knowledge graph embedding by double limit scoring loss. IEEE Trans Knowl Data Eng 1–1. https://doi.org/10.1109/TKDE.2021.3060755

  52. Zou X, Wang X, Cen S, Dai G, Liu C (2022) Knowledge graph embedding with self adaptive double-limited loss. Knowl-Based Syst 252:109310. https://doi.org/10.1016/j.knosys.2022.109310

    Article  Google Scholar 

  53. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. In: Advances in Neural Information Processing Systems, vol. 27

  54. Cai L, Wang WY (2018) Kbgan: Adversarial learning for knowledge graph embeddings. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). pp. 1470–1480. https://doi.org/10.18653/v1/N18-1133

  55. Perozzi B, Al-Rfou R, Skiena S (2014) Deepwalk: Online learning of social representations. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 701–710. https://doi.org/10.1145/2623330.2623732

  56. Tang Y, Huang J, Wang G, He X, Zhou B (2020) Orthogonal relation transforms with graph context modeling for knowledge graph embedding. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 2713–2722. https://doi.org/10.18653/v1/2020.acl-main.241

  57. Luong M-T, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. In: Proceedings of the 2015 Conference on EmpiricalMethods in Natural Language Processing. pp. 1412–1421

  58. Toutanova K, Chen D (2015) Observed versus latent features for knowledge base and text inference. In: Proceedings of the 3rd Workshop on Continuous Vector Space Models and Their Compositionality (CVSC). pp. 57–66

  59. Bordes A, Glorot X, Weston J, Bengio Y (2012) Joint learning of words and meaning representations for open-text semantic parsing. Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics 22:127–135

    Google Scholar 

  60. Han X, Cao S, Lv X, Lin Y, Liu Z, Sun M, Li J (2018) Openke: An open toolkit for knowledge embedding. In: Proceedings of EMNLP. pp. 139–144

Download references

Acknowledgements

This work was supported in part by the Postgraduate Innovation Development Fund Project of Shenzhen University, China (Grants 0000470814), in part by the National Natural Science Foundation of China (Grants 61976141, 61732011 and 62106148) and the Project funded by China Postdoctoral Science Foundation under Grant no. 2021M702259.

Author information

Authors and Affiliations

  1. College of Computer Science and Software Engineering, Shenzhen University, Shenzhen, 518060, China

    Si Cen, Xizhao Wang, Xiaoying Zou, Chao Liu & Guoquan Dai

  2. Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen University, Shenzhen, 518060, China

    Xizhao Wang

Authors
  1. Si Cen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xizhao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoying Zou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Guoquan Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xizhao Wang.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cen, S., Wang, X., Zou, X. et al. New attention strategy for negative sampling in knowledge graph embedding. Appl Intell 53, 26418–26438 (2023). https://doi.org/10.1007/s10489-023-04901-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-023-04901-0

Keywords

Search

Navigation