CAKE: A Context-Aware Knowledge Embedding Model of Knowledge Graph

Abstract

Recently, knowledge embedding on knowledge Graph (KG) has drawn increasing attention from both academia and industry for its concise rationale and promising prospects. However, performances of existing knowledge embedding methods are mostly either far from satisfactory, or exhibits weakness for generalization. In this work, a context-aware knowledge embedding model (CAKE) has been proposed for applications like knowledge completion and link prediction. We model the generative process of KG formation based on latent Dirichlet allocation and hierarchical Dirichlet process, where the latent semantic structure of knowledge elements is learned as contexts. Contextual information, i.e. the context-specific probability distribution over elements, is thereafter leveraged in a translation-based embedding model. Essentially, we develop loss function in a probabilistic style to approximately realize the “attention” mechanism in our model. In this work, the learned embeddings of entities and relations are applied to link prediction and triple classification in experiments and our model shows the best performance compared with multiple baselines.

A Appendix: Optimization of CAKE

For entity $\epsilon$, the gradient of its embedding is

$$\begin{aligned} \begin{aligned} \frac{1}{2}\frac{\partial \mathcal {L}}{\partial e^l_\epsilon }&= \sum _{c\in \mathcal {C}}\sum _{(h,r,t)\in \mathcal {D}^+}\sum _{(h',r',t')\in \mathcal {D}^-}\sum _{l'=1}^L(w_c^lw_c^{l'}-1)\{p_c(h,r,t)\\&\times [e_h^{l'}-e_t^{l'}-\omega _c^\top (e_h-e_t)\omega _c^{l'}+P_c^{l'}(e_r)]\times (I_{t=\epsilon }\\&-I_{h=\epsilon })-(\kappa -p_c(h',r',t'))\times [e_{h'}^{l'}-e_{t'}^{l'}\\&-\omega _c^\top (e_{h'}-e_{l'})\omega _c^{l'}+P_c^{l'}(e_{r'})]\times (I_{t'=\epsilon -I_{h'=\epsilon }})\} \end{aligned} \end{aligned}$$

For relation $\pi$, the gradient of its embedding is

$$\begin{aligned} \begin{aligned} \frac{1}{2}\frac{\partial \mathcal {L}}{\partial e^l_\pi }&= \sum _{c\in \mathcal {C}}\sum _{(h,r,t)\in \mathcal {D}^+}\sum _{(h',r',t')\in \mathcal {D}^-}\sum _{l'=1}^Lw_c^lw_c^{l'}\times \{I_{r=\pi }\\&\times p_c(h,r,t) \times [P_c^{l'}(e_h)+e_r^{l'}-(\omega _c^\top e_r)\omega _r^{l'}\\&-P_c^{l'}(e_t)]- I_{r'=\pi }\times (\kappa -p_c(h',r',t'))\\&\times [P_c^{l'}(e_{h'}+e_{r'}^{l'}-(\omega _c^\top e_{r'})\omega _{r'}^{l'}-P_c^{l'})]\} \end{aligned} \end{aligned}$$

For context $\delta$, the gradient of its normal vector is

$$\begin{aligned} \begin{aligned} \frac{1}{2}\frac{\partial \mathcal {L}}{\partial e^l_\pi }&= \sum _{c\in \mathcal {C}}\sum _{(h,r,t)\in \mathcal {D}^+}\sum _{(h',r',t')\in \mathcal {D}^-}\sum _{l'=1}^L(e_t^l-e_h^l-e_r^l)\times \{I_{c=\delta }\\&\times p_c(h,r,t)\times (\omega _c^{l'}+I_{l=l'}\times \omega _c^l)\times [e_h^{l'}+e_r^{l'}-e_t^{l'}\\&-\omega _c^\top (e_h+e_r+e_t)\omega _c^{l'}]-I_{c'=\epsilon }\times (\kappa -p_c'(h',r',t'))\\&\times (\omega _{c'}^{l'}+I_{l=l'}\times \omega _{c'}^l)\times [e_h^{l'}+e_r^{l'}-e_t^{l'}\\&-\omega _{c'}^\top (e_h+e_r+e_t)\omega _{c'}^{l'}]\} \end{aligned} \end{aligned}$$