Skip to main content
SpringerLink
Log in

Knowledge Base Completion by Variational Bayesian Neural Tensor Decomposition

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Knowledge base completion is an important research problem in knowledge bases, which play important roles in question answering, information retrieval, and other applications. A number of relational learning algorithms have been proposed to solve this problem. However, despite their success in modeling the entity relations, they are not well founded in a Bayesian manner and thus are hard to model the prior information of the entity and relation factors. Furthermore, they under-represent the interaction between entity and relation factors. In order to avoid these disadvantages, we provide a neural-inspired approach, namely Bayesian Neural Tensor Decomposition approach for knowledge base completion based on the Stochastic Gradient Variational Bayesian framework. We employ a multivariate Bernoulli likelihood function to represent the existence of facts in knowledge graphs. We further employ a Multi-layered Perceptrons to represent more complex interactions between the latent subject, predicate, and object factors. The SGVB framework can enable us to make efficient approximate variational inference for the proposed nonlinear probabilistic tensor decomposition by a novel local reparameterization trick. This way avoids the need of expensive iterative inference schemes such as MCMC and does not make any over-simplified assumptions about the posterior distributions, in contrary to the common variational inference. In order to evaluate the proposed model, we have conducted experiments on real-world knowledge bases, i.e., FreeBase and WordNet. Experimental results have indicated the promising performance of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Auer S, Bizer C, Kobilarov G, Lehmann J, Ives Z. DBpedia: A nucleus for a web of open data. In: Proceedings ISWC; 2007. p. 11–15.

  2. Bishop CM, Nasrabadi NM. Pattern recognition and machine learning. Springer, 2006. p. 461–462.

  3. Bollacker K, Evans C, Paritosh P, Sturge T, Taylor J. Freebase: a collaboratively created graph database for structuring human knowledge. In: ACM’S special interest group on management of data conference; 2008. p. 1247–1250.

  4. Bordes A, Glorot X, Weston J, Bengio Y. A semantic matching energy function for learning with multi-relational data - application to word-sense disambiguation. Mach Learn 2014;94(2):233–259.

    Article  Google Scholar 

  5. Bordes A, Usunier N, García-Durán A, Weston J, Yakhnenko O. Translating embeddings for modeling multi-relational data. In: Advances in neural information processing systems; 2013. pp. 2787–2795.

  6. Chen S, Lyu MR, King I, Xu Z. Exact and stable recovery of pairwise interaction tensors. In: Advances in neural information processing systems 26: 27th annual conference on neural information processing systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe; 2013. pp. 1691–1699.

  7. Dong X, Gabrilovich E, Heitz G, Horn W, Lao N, Murphy K, Strohmann T, Sun S, Zhang W. Knowledge vault: a web-scale approach to probabilistic knowledge fusion. In: ACM SIGKDD international conference on knowledge discovery and data mining; 2014. p. 601–610.

  8. Duchi J, Hazan E, Singer Y. Adaptive subgradient methods for online learning and stochastic optimization. J Mach Learn Res 2011;12(Jul):2121–2159.

    Google Scholar 

  9. Fan M, Zhou Q, Abel A, Zheng TF, Grishman R. Probabilistic belief embedding for large-scale knowledge population. Cogn Comput 2016;8(6):1087–1102.

    Article  Google Scholar 

  10. Huang S, Wang H, Li T, Li T, Xu Z. Robust graph regularized nonnegative matrix factorization for clustering. Data Min Knowl Discov 2018;32(2):483–503. https://doi.org/10.1007/s10618-017-0543-9.

    Article  Google Scholar 

  11. Huang S, Xu Z, Lv J. Adaptive local structure learning for document co-clustering. Knowl-Based Syst 2018;148:74–84. https://doi.org/10.1016/j.knosys.2018.02.020.

    Article  Google Scholar 

  12. Kingma DP, Welling M. 2013. Auto-encoding variational bayes. arXiv:1312.6114.

  13. Lao N, Cohen WW. Relational retrieval using a combination of path-constrained random walks. Mach Learn 2010;81(1):53–67.

    Article  Google Scholar 

  14. Lao N, Mitchell T, Cohen WW. Random walk inference and learning in a large scale knowledge base. In: Conference on empirical methods in natural language processing, EMNLP 2011, john mcintyre conference centre, edinburgh, uk, a meeting of sigdat, a special interest group of the ACL; 2012. p. 529–539.

  15. Li G, Xu Z, Wang L, Ye J, King I, Lyu MR. Simple and efficient parallelization for probabilistic temporal tensor factorization. In: 2017 international joint conference on neural networks, IJCNN 2017, anchorage; 2017, p. 1–8.

  16. Lin Y, Liu Z, Zhu X, Zhu X, Zhu X. Learning entity and relation embeddings for knowledge graph completion. In: Twenty-ninth AAAI conference on artificial intelligence; 2015. p. 2181– 2187.

  17. Liu B, Li Y, Xu Z. Manifold regularized matrix completion for multi-label learning with ADMM. Neural Netw 2018;101:57–67. https://doi.org/10.1016/j.neunet.2018.01.011.

    Article  PubMed  Google Scholar 

  18. Miller GA. Wordnet: a lexical database for english. Commun Acm 1995;38(11):39–41.

    Article  Google Scholar 

  19. Nickel M, Murphy K, Tresp V, Gabrilovich E. A review of relational machine learning for knowledge graphs. Proc IEEE 2016;104(1):11–33.

    Article  Google Scholar 

  20. Nickel M, Tresp V. 2013. Logistic tensor factorization for multi-relational data. arXiv:1306.2084.

  21. Nickel M, Tresp V, Kriegel HP. A three-way model for collective learning on multi-relational data. In: International conference on international conference on machine learning; 2011, vol. 11. p. 809–816.

  22. Ofek N, Poria S, Rokach L, Cambria E, Hussain A, Shabtai A. Unsupervised commonsense knowledge enrichment for domain-specific sentiment analysis. Cogn Comput 2016;8(3):467–477.

    Article  Google Scholar 

  23. Socher R, Chen D, Manning CD, Ng AY. Reasoning with neural tensor networks for knowledge base completion. In: Advances in neural information processing systems; 2013. p. 926– 934.

  24. Suchanek FM, Kasneci G, Weikum G. Yago: a core of semantic knowledge. Proceedings of the 16th international conference on World Wide Web. ACM; 2007. p. 697–706.

  25. Sutskever I, Salakhutdinov R, Tenenbaum JB. Modelling relational data using bayesian clustered tensor factorization. In: Advances in neural information processing systems; 2009. p. 1821–1828.

  26. Wang QF, Cambria E, Liu CL, Hussain A. Common sense knowledge for handwritten Chinese text recognition. Cogn Comput 2013;5(2):234–242.

    Article  Google Scholar 

  27. Wang Z, Zhang J, Feng J, Chen Z. Knowledge graph embedding by translating on hyperplanes. In: The association for the advance of artificial intelligence; 2014, vol. 14. p. 1112–1119.

  28. Weston J, Bordes A, Yakhnenko O, Usunier N. Connecting language and knowledge bases with embedding models for relation extraction. In: Conference on empirical methods in natural language processing; 2013. p. 1366–1371.

  29. Xu Z, Yan F, Qi Y. Infinite tucker decomposition: Nonparametric bayesian models for multiway data analysis. In: Proceedings of the 29th international conference on machine learning, ICML 2012. Edinburgh; 2012.

  30. Xu Z, Yan F, Qi Y. Bayesian nonparametric models for multiway data analysis. IEEE Trans Pattern Anal Mach Intell 2015;37(2):475–487.

    Article  PubMed  Google Scholar 

  31. Yang X, Huang K, Zhang R, Hussain A. Learning latent features with infinite non-negative binary matrix tri-factorization. IEEE Trans Emerg Topics Comput Intell. 2018;2(3). https://doi.org/10.1109/TETCI.2018.2806934.

  32. Zhe S, Qi Y, Park Y, Xu Z, Molloy I, Chari S. Dintucker: Scaling up gaussian process models on large multidimensional arrays. In: Proceedings of the thirtieth AAAI conference on artificial intelligence. Phoenix; 2016. p. 2386–2392.

  33. Zhe S, Xu Z, Chu X, Qi Y, Park Y. Scalable nonparametric multiway data analysis. In: Proceedings of the eighteenth international conference on artificial intelligence and statistics, AISTATS 2015, San Diego; 2015.

  34. Zhe S, Zhang K, Wang P, Lee K, Xu Z, Qi Y, Ghahramani Z. Distributed flexible nonlinear tensor factorization. In: Advances in neural information processing systems 29, Barcelona; 2016. p. 920–928.

  35. Zhong G, Cheriet M. Large margin low rank tensor analysis. Neural Comput 2014;26(4):761–780.

    Article  PubMed  Google Scholar 

  36. Zhong G, Cheriet M. Tensor representation learning based image patch analysis for text identification and recognition. Pattern Recogn 2015;48(4):1211–1224.

    Article  Google Scholar 

  37. Zhu J. Max-margin nonparametric latent feature models for link prediction. In: Proceedings of the 29th international coference on international conference on machine learning. Omnipress; 2012. p. 1179–1186.

Download references

Funding

All the authors except Yafang Wang were supported by the Natural Science Foundation of China (No. 61572111), 1000-Talent Startup Funds (Nos. G05QNQR004, A1098531023601041) and a Fundamental Research Fund for the Central Universities of China (No. ZYGX2016Z003).

Author information

Authors and Affiliations

  1. SMILE Lab, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, Sichuan, China

    Lirong He, Bin Liu, Guangxi Li, Yongpan Sheng & Zenglin Xu

  2. Ant Financial Service Co., Hangzhou, Zhejiang, China

    Yafang Wang

Authors
  1. Lirong He
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guangxi Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yongpan Sheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yafang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Zenglin Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zenglin Xu.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed Consent

Informed consent was not required as no humans or animals were involved.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

He, L., Liu, B., Li, G. et al. Knowledge Base Completion by Variational Bayesian Neural Tensor Decomposition. Cogn Comput 10, 1075–1084 (2018). https://doi.org/10.1007/s12559-018-9565-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-018-9565-x

Keywords

Search

Navigation