Cluster Robust Inference for Embedding-Based Knowledge Graph Completion

Abstract

Knowledge Graphs (KGs) are able to structure and represent knowledge in complex systems, whereby their completeness impacts the quality of any further application. Real world KGs are notoriously incomplete, which is why KG Completion (KGC) methods emerged. Most KGC methods rely on Knowledge Graph Embedding (KGE) based link prediction models, which provide completion candidates for a sparse KG. Metrics like the Mean Rank, the Mean Reciprocal Rank or Hits@K evaluate the quality of those models, like TransR, DistMult or ComplEx. Based on the principle of supervised learning, these metrics evaluate a KGC model trained on a training dataset, based on the partition of true completion candidates it achieves on a test dataset. Dealing with real world, complex KGs, we found that sparsity is not equally distributed across a KG, but rather grouped in clusters. We use modularity-based KG clustering, to approximate sparsity levels in a KG. Furthermore, we postulate that prediction errors of an embedding-based KGC model are correlated within clusters of a KG but uncorrelated between them and formalize a new, cluster-robust KGC evaluation metric. We test our metric using six benchmark dataset and one real-world industrial example dataset. Our experiments show its superiority to existing metrics with regards to the prediction of cluster-robust triplets$^{1}$($^{1}$The code is available at https://github.com/simoncharmms/crmrr.).

Appendix

1.1 Appendix A: Relevant Notations of this Paper

Notation	Meaning
G	A graph
$\lbrace (h, r, t) \rbrace \in \mathcal {T}$	A set of triples with heads, relations and tails
$e \in \mathcal {E}$	A set of entities e
$r \in \mathcal {R}$	A set of relations r
$R_r \in \mathbb {R}^{d \times d}$	A $d \times d$ dimensional matrix with coefficients $a_{ji}$
$r_r \in \mathbb {R}^{d}$	A d-dimensional vector
$k^{out}=\sum _j a_{ji}$	The outbound degree
$k_i^{in}=\sum _j a_{ij}$	The inbound degree
Deg	The total degree of a graph
T(e)	The number of triangles through entity e
Trans	The transitivity of a graph
CCO	The mean clustering coefficient of a graph
$r_{cc}$	The rank of a cross-cluster completion candidate
n	The total count of ranks $r_{i}$
m	The count of cross-cluster ranks $r_{j \vert cc}$ from $\lbrace (h,r,t)_{cc} \rbrace$
$|c |$	The count of clusters of a clustered KG
$(h,r,t)_{cc}$	A cross-cluster completion candidate

1.2 Appendix B: Metrics of Used Knowledge Graphs

	Industry KG	$BTC_{alpha}$	$BTC_{otc}$	$web-google$
		[14]	[13]	[16]
Node count	116699	24185	35591	5105039
Relation count	5710	3782	5881	875713
Deg	40.875	12.789	12.103	11.659
Trans	0.012	0.063	0.045	0.449
CCO	0.044	0.158	0.151	0.369
$|c |$	61	79	87	29722
	$sx-stackoverflow$	$web-amazon$	$web-facebook$
	[24]	[22]	[27]
Node count	574795	925872	171002
Relation count	406972	334863	22470
Deg	2.824	5.529	15.220
Trans	0.001	0.103	0.124
CCO	0.001	0.198	0.179
$|c |$	90266	31916	790

1.3 Appendix C: Training Details and Results by KG and Model.

	Model	Epochs	Batch size	m/n	MR	MRR	$H_{10}$	CRMRR
$web-google$	TransR			0.381	6.070	0.142	0.887	0.221
	DistMult	510	32	0.137	49.370	0.024	0.362	0.389
	ComplEx			0.152	315.440	0.001	0.135	0.229
$web-amazon$	TransR			0.212	11.300	0.093	0.362	0.116
	DistMult	510	32	0.365	29.620	0.012	0.566	0.217
	ComplEx			0.195	508.480	0.009	0.112	0.368
$sx-stackoverflow$	TransR			0.262	7.150	0.074	0.791	0.145
	DistMult	510	32	0.165	38.400	0.025	0.498	0.285
	ComplEx			0.265	484.940	0.002	0.255	0.115
$web-facebook$	TransR			0.185	3.660	0.156	0.272	0.083
	DistMult	510	32	0.114	26.880	0.008	0.798	0.127
	ComplEx			0.098	795.670	0.001	0.302	0.136
Industry KG	TransR			0.134	8.312	0.124	0.727	0.189
	DistMult	510	32	0.119	54.854	0.018	0.578	0.283
	ComplEx			0.219	470.813	0.002	0.184	0.171
$BTC_{alpha}$	TransR			0.530	7.400	0.056	0.210	0.082
	DistMult	510	32	0.262	20.840	0.011	0.438	0.180
	ComplEx			0.235	404.900	0.004	0.076	0.119
$BTC_{otc}$	TransR			0.154	3.820	0.113	0.626	0.155
	DistMult	510	32	0.170	21.390	0.012	0.394	0.202
	ComplEx			0.32	348.40	0.001	0.134	0.081