skip to main content
10.1145/3480001.3480022acmotherconferencesArticle/Chapter ViewAbstractPublication PagesicdltConference Proceedingsconference-collections
research-article

Deep Learning For Knowledge Graph Completion With XLNET

Published: 12 November 2021 Publication History

Abstract

Knowledge Graph is a graph knowledge base composed of fact entities and relations. Recently, the adoption of Knowledge Graph in Natural Language Processing tasks has proved the efficiency and convenience of KG. Therefore, the plausibility of Knowledge Graph become an import subject, which is also named as KG Completion or Link Prediction. The plausibility of Knowledge Graph reflects in the validness of triples which is structured representation of the entities and relations of Knowledge Graph. Some research work has devoted to KG Completion tasks. The typical methods include semantic matching models like TransE or TransH and Pre-trained models like KG-BERT. In this article, we propose a novel method based on the pre-trained model XLNET and the classification model to verify whether the triples of Knowledge Graph are valid or not. This method takes description of entities or relations as the input sentence text for fine-tuning. Meanwhile contextualized representations with rich semantic information can be obtained by XLNET, avoiding limitations and shortcomings of other typical neural network models. Then these representations are fed into a classifier for classification. Experimental results show that there is an improvement in KG Completion Tasks that the proposed method has achieved.

References

[1]
Bordes A, Usunier N, Garcia-Duran A, Translating embeddings for modeling multi-relational data[C]//Neural Information Processing Systems (NIPS). 2013: 1-9.
[2]
Wang Z, Zhang J, Feng J, Knowledge graph embedding by translating on hyperplanes[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2014, 28(1).
[3]
Yang B, Yih W, He X, Embedding entities and relations for learning and inference in knowledge bases[J]. arXiv preprint arXiv:1412.6575, 2014.
[4]
Wang Q, Mao Z, Wang B, Knowledge graph embedding: A survey of approaches and applications[J]. IEEE Transactions on Knowledge and Data Engineering, 2017, 29(12): 2724-2743.
[5]
Peters, Matthew E., "Deep contextualized word representations." arXiv preprint arXiv:1802.05365 (2018).
[6]
Radford A, Narasimhan K, Salimans T, Improving language understanding by generative pre-training[J]. 2018.
[7]
Devlin J, Chang M W, Lee K, Bert: Pre-training of deep bidirectional transformers for language understanding[J]. arXiv preprint arXiv:1810.04805, 2018.
[8]
Yao L, Mao C, Luo Y. KG-BERT: BERT for knowledge graph completion[J]. arXiv preprint arXiv:1909.03193, 2019.
[9]
Mikolov T, Chen K, Corrado G, Efficient estimation of word representations in vector space[J]. arXiv preprint arXiv:1301.3781, 2013
[10]
I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in NIPS, 2014.
[11]
Vaswani A, Shazeer N, Parmar N, Attention is all you need[J]. arXiv preprint arXiv:1706.03762, 2017.
[12]
Mnih V, Heess N, Graves A. Recurrent models of visual attention[C]//Advances in neural information processing systems. 2014: 2204-2212.
[13]
Bahdanau D, Cho K, Bengio Y, Neural Machine Translation by Jointly Learning to Align and Translatep[C]. international conference on learning representations, 2015.
[14]
Yang Z, Dai Z, Yang Y, XLNet: Generalized autoregressive pretraining for language understanding[C]//Advances in neural information processing systems. 2019: 5754-5764.
[15]
Peters M E, Neuman M, Iyyer M, DEEP CONTEXTUALIZED WORD REPRESENTATIONS[C]. north american chapter of the association for computational linguistics, 2018: 2227-2237.
[16]
Radford A, Narasimhan K, Salimans T, Improving language understanding by generative pre-traing[J]. URL https://s3-us-west-2.amazonaws.com/openai-assets/researchcovers/languageunsupervised/language understading paper.pdf, 2018.
[17]
Horev R. BERT Explained: State of the art language model for NLP[J]. Towards Data Science, Nov, 2018, 10.
[18]
Shin J, Lee Y, Jung K. Effective sentence scoring method using bert for speech recognition[C]//Asian Conference on Machine Learning. PMLR, 2019: 1081-1093.
[19]
Jeong C, Jang S, Park E, A context-aware citation recommendation model with BERT and graph convolutional networks[J]. Scientometrics, 2020, 124(3): 1907-1922.
[20]
Berners-Lee T, Hendler J, Lassila O. The semantic web[J]. Scientific american, 2001, 284(5): 34-43.
[21]
Amit S. Introducing the Knowledge Graph: things, not strings[R]. Official Google Blog, 2012.
[22]
Jiyeon Kim, Youngchang Kim, Hyesun Suh, and Jongjin Jung, "Diversity of Recommendation with Considering Data Similarity among Different Types of Contents," Vol. 7, No. 2, pp. 76-80, May, 2016.
[23]
Subhash K. Shinde and Uday V. Kulkarni, "Hybrid Personalized Recommender System Using Fast K-medoids Clustering Algorithm," Journal of Advances in Information Technology, Vol. 2, No. 3, pp. 152-158, August, 2011.
[24]
Mukesh Kumar and Renu Vig, "Multilingual Context Ontology Rule Enhanced Focused Web Crawler," Journal of Advances in Information Technology, Vol. 1, No. 1, pp. 21-25, February, 2010.
[25]
Shao-Fang Wen and Basel Katt, "Development of Ontology-Based Software Security Learning System with Contextualized Learning Approach," Journal of Advances in Information Technology, Vol. 10, No. 3, pp. 81-90, August 2019.
[26]
Peilu Wang, Hao Jiang, Jingfang Xu, Qi Zhang; Knowledge Graph Construction and Applications for Web Search and Beyond. Data Intelligence 2019; 1 (4): 333–349.
[27]
Auer S, Bizer C, Kobilarov G, Dbpedia: A nucleus for a web of open data[M]//The semantic web. Springer, Berlin, Heidelberg, 2007: 722-735.
[28]
Bollacker K, Evans C, Paritosh P, Freebase: a collaboratively created graph database for structuring human knowledge[C]//Proceedings of the 2008 ACM SIGMOD international conference on Management of data. 2008: 1247-1250.
[29]
Miller G A. WordNet: a lexical database for English[J]. Communications of the ACM, 1995, 38(11): 39-41.
[30]
Suchanek F M, Kasneci G, Weikum G. Yago: a core of semantic knowledge[C]//Proceedings of the 16th international conference on World Wide Web. 2007: 697-706.
[31]
Liu H, Singh P. ConceptNet—a practical commonsense reasoning tool-kit[J]. BT technology journal, 2004, 22(4): 211-226.
[32]
Socher R, Chen D, Manning C D, Reasoning with neural tensor networks for knowledge base completion[C]//Advances in neural information processing systems. 2013: 926-934.
[33]
Dettmers T, Minervini P, Stenetorp P, Convolutional 2d knowledge graph embeddings[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2018, 32(1).
[34]
Chung J, Gulcehre C, Cho K H, Empirical evaluation of gated recurrent neural networks on sequence modeling[J]. arXiv preprint arXiv:1412.3555, 2014.

Cited By

View all
  • (2023)Research on semantic representation and citation recommendation of scientific papers with multiple semantics fusionScientometrics10.1007/s11192-022-04566-5128:2(1367-1393)Online publication date: 3-Jan-2023
  • (2022)Medical Knowledge Graph Completion Based on Word EmbeddingsInformation10.3390/info1304020513:4(205)Online publication date: 18-Apr-2022
  1. Deep Learning For Knowledge Graph Completion With XLNET

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    ICDLT '21: Proceedings of the 2021 5th International Conference on Deep Learning Technologies
    July 2021
    131 pages
    ISBN:9781450390163
    DOI:10.1145/3480001
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 12 November 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. GRU
    2. KG Completion
    3. Knowledge Graph
    4. LSTM
    5. XLNet

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Conference

    ICDLT 2021

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)10
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 18 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)Research on semantic representation and citation recommendation of scientific papers with multiple semantics fusionScientometrics10.1007/s11192-022-04566-5128:2(1367-1393)Online publication date: 3-Jan-2023
    • (2022)Medical Knowledge Graph Completion Based on Word EmbeddingsInformation10.3390/info1304020513:4(205)Online publication date: 18-Apr-2022

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media