A Closer Look at Probability Calibration of Knowledge Graph Embedding
Authors: 
Ruiqi Zhu, Fangrong Wang, Alan Bundy, Xue Li, Kwabena Nuamah, Lei Xu, Stefano Mauceri, Jeff Z. PanAuthors Info & Claims
Ruiqi Zhu, Fangrong Wang, Alan Bundy, Xue Li, Kwabena Nuamah, Lei Xu, Stefano Mauceri, Jeff Z. PanAuthors Info & ClaimsPages 104 - 109
Abstract
When the estimated probabilities do not match the relative frequencies, we say these estimated probabilities are uncalibrated [39], which may cause incorrect decision making, and is particularly undesired in high-stakes tasks [45]. Knowledge Graph embedding models are reported to produce uncalibrated probabilities [36], e.g., for all the triples predicted with probability 0.9, the percentage of them being truly correct triples is not . In this article, we take a closer look at this problem. First, we confirmed the issue that typical KG Embedding models are uncalibrated. Then, we show how off-the-shelf calibration techniques can be used to mitigate this issue, among which binning-based calibration produces more calibrated probabilities. We also investigated the possible reasons for the uncalibrated probabilities and found that the expit transform, the way used to convert embedding scores into probabilities, is ineffective in most cases.
References
[1]
Ralph Abboud, Ismail Ceylan, Thomas Lukasiewicz, and Tommaso Salvatori. 2020. Boxe: A box embedding model for knowledge base completion. Advances in Neural Information Processing Systems 33 (2020), 9649–9661.
[2]
Mehdi Ali, Max Berrendorf, Charles Tapley Hoyt, Laurent Vermue, Sahand Sharifzadeh, Volker Tresp, and Jens Lehmann. 2021. PyKEEN 1.0: A Python Library for Training and Evaluating Knowledge Graph Embeddings. Journal of Machine Learning Research 22, 82 (2021), 1–6. http://jmlr.org/papers/v22/20-825.html
[3]
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems 26 (2013).
[4]
Hongyun Cai, Vincent W Zheng, and Kevin Chen-Chuan Chang. 2018. A comprehensive survey of graph embedding: Problems, techniques, and applications. IEEE Transactions on Knowledge and Data Engineering 30, 9(2018), 1616–1637.
[5]
Zongsheng Cao, Qianqian Xu, Zhiyong Yang, Xiaochun Cao, and Qingming Huang. 2021. Dual quaternion knowledge graph embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 6894–6902.
[6]
Linlin Chao, Jianshan He, Taifeng Wang, and Wei Chu. 2021. PairRE: Knowledge Graph Embeddings via Paired Relation Vectors. (Aug. 2021), 4360–4369. https://doi.org/10.18653/v1/2021.acl-long.336
[7]
Luca Costabello, Sumit Pai, Chan Le Van, Rory McGrath, Nicholas McCarthy, and Pedro Tabacof. 2019. AmpliGraph: a Library for Representation Learning on Knowledge Graphs. https://doi.org/10.5281/zenodo.2595043
[8]
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2d knowledge graph embeddings. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32.
[9]
Jhonatan Garcia, Jeff Z. Pan, Achille Fokoue, Katia Sycara, Yuqing Tang, and Federico Cerutti.2015. Handling uncertainty: An extension of DL-Lite with Subjective Logic. In Proc. of 28th International Workshop on Description Logics (DL 2015).
[10]
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning. PMLR, 1321–1330.
[11]
Lars Holmberg and Andrew Vickers. 2013. Evaluation of prediction models for decision-making: beyond calibration and discrimination. PLoS medicine 10, 7 (2013), e1001491.
[12]
Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge graph embedding via dynamic mapping matrix. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: Long papers). 687–696.
[13]
Charles Kemp, Joshua B Tenenbaum, Thomas L Griffiths, Takeshi Yamada, and Naonori Ueda. 2006. Learning systems of concepts with an infinite relational model. In AAAI, Vol. 3. 5.
[14]
Meelis Kull, Telmo Silva Filho, and Peter Flach. 2017. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Artificial Intelligence and Statistics. PMLR, 623–631.
[15]
Meelis Kull, Telmo M Silva Filho, and Peter Flach. 2017. Beyond sigmoids: How to obtain well-calibrated probabilities from binary classifiers with beta calibration. Electronic Journal of Statistics 11, 2 (2017), 5052–5080.
[16]
Fabian Kuppers, Jan Kronenberger, Amirhossein Shantia, and Anselm Haselhoff. 2020. Multivariate confidence calibration for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 326–327.
[17]
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In Twenty-ninth AAAI conference on artificial intelligence.
[18]
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence.
[19]
Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. 2015. A review of relational machine learning for knowledge graphs. Proc. IEEE 104, 1 (2015), 11–33.
[20]
Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio. 2016. Holographic embeddings of knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 30.
[21]
Alexandru Niculescu-Mizil and Rich Caruana. 2005. Predicting good probabilities with supervised learning. In Proceedings of the 22nd international conference on Machine learning. 625–632.
[22]
J.Z. Pan, G. Vetere, J.M. Gomez-Perez, and H. Wu (Eds.). 2017. Exploiting Linked Data and Knowledge Graphs for Large Organisations. Springer.
[23]
Jeff Z. Pan. 2009. Resource Description Framework. In Handbook on Ontologies. 71–90. https://doi.org/10.1007/978-3-540-92673-3_3
[24]
Jeff Z. Pan, Giorgos Stamou, Vassilis Tzouvaras, and Ian Horrocks. 2005. f-SWRL: A Fuzzy Extension of SWRL. In Proc. of the International Conference on Artificial Neural Networks (ICANN 2005).
[25]
Pouya Pezeshkpour, Yifan Tian, and Sameer Singh. 2020. Revisiting evaluation of knowledge base completion models. In Automated Knowledge Base Construction.
[26]
John Platt 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers 10, 3 (1999), 61–74.
[27]
Guilin Qi, Jeff Z. Pan, and Qiu Ji. 2007. A Possibilistic Extension of Description Logics. In Proc. of 2007 International Workshop on Description Logics (DL2007).
[28]
Guilin Qi, Jeff Z. Pan, and Qiu Ji. 2007. Extending Description Logics with Uncertainty Reasoning in Possibilistic Logic. In the Proc. of the 9th European Conference on Symbolic and Quantitave Approaches to Reasoning with Uncertainty (ECSQARU’2007). 828–839.
[29]
Aishwarya Rao. 2021. Calibrating Knowledge Graphs. Rochester Institute of Technology.
[30]
Tara Safavi, Danai Koutra, and Edgar Meij. 2020. Evaluating the Calibration of Knowledge Graph Embeddings for Trustworthy Link Prediction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 8308–8321.
[31]
Murat Sensoy, Jeff Z. Pan, Achille Fokoue, Mudhakar Srivatsa, and Felipe Meneguzzi. 2012. Using Subjective Logic to Handle Uncertainty and Conflicts. In Proc. of the 2012 International Symposium on Advances in Trusted and Secure Information Systems (TrustCom 2012).
[32]
Baoxu Shi and Tim Weninger. 2018. Open-world knowledge graph completion. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32.
[33]
Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. Advances in neural information processing systems 26 (2013).
[34]
Giorgos Stoilos, Giorgos B. Stamou, and Jeff Z. Pan. 2006. Handling Imprecise Knowledge with Fuzzy Description Logic. In Proceedings of the 2006 International Workshop on Description Logics (DL2006).
[35]
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. arXiv preprint arXiv:1902.10197(2019).
[36]
Pedro Tabacof and Luca Costabello. 2020. Probability Calibration for Knowledge Graph Embedding Models. In International Conference on Learning Representations. https://openreview.net/forum?id=S1g8K1BFwS
[37]
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In International conference on machine learning. PMLR, 2071–2080.
[38]
Ben Van Calster and Andrew J Vickers. 2015. Calibration of risk prediction models: impact on decision-analytic performance. Medical decision making 35, 2 (2015), 162–169.
[39]
Bas C Van Fraassen. 1983. Calibration: A frequency justification for personal probability. In Physics, philosophy and psychoanalysis. Springer, 295–319.
[40]
Susan Vineberg. 2016. Dutch Book Arguments. In The Stanford Encyclopedia of Philosophy (Spring 2016 ed.), Edward N. Zalta (Ed.). Metaphysics Research Lab, Stanford University.
[41]
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the AAAI conference on artificial intelligence, Vol. 28.
[42]
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2014. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575(2014).
[43]
Bianca Zadrozny and Charles Elkan. 2001. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml, Vol. 1. Citeseer, 609–616.
[44]
Xiangxiang Zeng, Xinqi Tu, Yuansheng Liu, Xiangzheng Fu, and Yansen Su. 2022. Toward better drug discovery with knowledge graph. Current opinion in structural biology 72 (2022), 114–126.
[45]
Shengjia Zhao, Michael Kim, Roshni Sahoo, Tengyu Ma, and Stefano Ermon. 2021. Calibrating predictions to decisions: A novel approach to multi-class calibration. Advances in Neural Information Processing Systems 34 (2021), 22313–22324.
[46]
Zhehui Zhou, Can Wang, Yan Feng, and Defang Chen. 2022. JointE: Jointly utilizing 1D and 2D convolution for knowledge graph embedding. Knowledge-Based Systems 240 (2022), 108100. https://doi.org/10.1016/j.knosys.2021.108100
Index Terms
- A Closer Look at Probability Calibration of Knowledge Graph Embedding
Index terms have been assigned to the content through auto-classification.
Recommendations
Knowledge Graph Embedding: A Locally and Temporally Adaptive Translation-Based Approach
A knowledge graph is a graph with entities of different types as nodes and various relations among them as edges. The construction of knowledge graphs in the past decades facilitates many applications, such as link prediction, web search analysis, ...
FedE: Embedding Knowledge Graphs in Federated Setting
IJCKG '21: Proceedings of the 10th International Joint Conference on Knowledge GraphsKnowledge graphs (KGs) become widespread and many organizations construct as well as maintain their own knowledge graphs. Same as the data isolation which has been a long-standing problem, knowledge graph isolation is common in real knowledge graph ...
Knowledge Graph Embedding with Diversity of Structures
WWW '17 Companion: Proceedings of the 26th International Conference on World Wide Web CompanionIn recent years, different web knowledge graphs, both free and commercial, have been created. Knowledge graphs use relations between entities to describe facts in the world. We engage in embedding a large scale knowledge graph into a continuous vector ...
Comments
Information & Contributors
Information
Published In
October 2022
134 pages
ISBN:9781450399876
DOI:10.1145/3579051
Copyright © 2022 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 13 February 2023
Check for updates
Author Tags
Qualifiers
- Extended-abstract
- Research
- Refereed limited
Conference
IJCKG 2022
IJCKG 2022: 11th International Joint Conference On Knowledge Graphs
October 27 - 28, 2022
Hangzhou, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 85Total Downloads
- Downloads (Last 12 months)31
- Downloads (Last 6 weeks)1
Reflects downloads up to 28 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format