Link prediction using deep autoencoder-like non-negative matrix factorization with L21-norm

Abstract

Link prediction aims to predict missing links or eliminate spurious links and anticipate new links by analyzing observed network topological structure information. Non-negative matrix factorization(NMF) is widely used in solving the issue of link prediction due to its good interpretability and scalability. However, most existing NMF-based approaches involve shallow decoder models, which are incapable of capturing complex hierarchical information hidden in networks, and seldom consider random noise. To address these issues, a novel deep autoencoder-like nonnegative matrix factorization method with $\varvec{L_{2,1}}$ norm for link prediction models(DANMFL) is proposed. Unlike conventional NMF-based approaches, our model contains a decoder component and an encoder component, which capture complex hierarchical information effectively, leading to more accurate prediction results. In addition, we use the $\varvec{L_{2,1}}$ norm to remove random noise in each layer and the convergence of our model is strictly proven. We conduct extensive experiments on twelve real-world networks and the experimental results show that DANMFL is superior to existing state-of-the-art baseline approaches in terms of prediction accuracy. Codes are available at https://github.com/litongf/DANMFL.

Appendix A: Derivation for Eq. (17) and Eq. (22)

In this part, we will analyse the convergence of DANMFL and give the theoretical proof. Next, we list the following theorems:

Theorem 1

Updating W using the rule of (17) while fixing the rest variables, the objective function L(W) monotonically decreases.

Theorem 2

Updating V using the rule of (22) while fixing the rest variables, the objective function L(V) monotonically decreases.

We utilize auxiliary function [54] to help us prove theorems 1 and 2.

Definition 3.1

$G(h,h')$ is an auxiliary function for F(h) if the conditions in (A1) are satisfied.

$$\begin{aligned} G(h,h')\ge F(h),\ \ G(h,h)=F(h) \end{aligned}$$

(A1)

The function is really helpful by reason of the following lemma.

Lemma 3.2

   If G is an auxiliary function of L, then L is non-increasing under the update

$$\begin{aligned} h^{t+1}=\underset{h}{\arg \min }\ G(h,h^{t}) \end{aligned}$$

(A2)

Proof

$L(h^{t+1})\le G(h^{t+1},h^{t})\le G(h^{t},h^{t})=L(h^{t})$ $\square$

Lemma 3.3

Let $A\in \mathbb {R}^{n\times n}$ is an any nonnegative matrix, $B\in \mathbb {R}^{k\times k}$, $S \in \mathbb {R}^{n\times k}$, $S'\in \mathbb {R}^{n\times k}$, A and B are symmetric, then the following inequality holds [62]:

$$\begin{aligned} \sum _{i=1}^{n}\sum _{j=1}^{k}\frac{(AS'B)_{ij}S_{ij}^2}{S'_{ij}}\ge Tr(S^TASB) \end{aligned}$$

(A3)

Here, we firstly prove the theorem 1 and rewrite the objective function L with removing unrelated elements.

$$\begin{aligned} L(W_{i})= & {} Tr(-2\Phi _{i+1}V_{i}X^T \Psi _{i-1}W_{i}+\Psi _{i-1}^T\Psi _{i-1}W_{i}\Phi _{i+1}\nonumber \\{} & {} \times V_{i}V_{i}^T\Phi _{i+1}^TW_{i}^T) +Tr(-2\Psi _{i-1}^T X V_{i}^T\Phi _{i+1}^T W_{i}^T\nonumber \\{} & {} +\Phi _{i+1}\Phi _{i+1}^TW_{i}^T\Psi _{i-1}^T X X^T\Psi _{i-1} W_{i}) \end{aligned}$$

(A4)

According to the formulas: $Tr(A^T)=Tr(A)$ and $Tr(AB)$ $=Tr(BA)$, the (A4) can be further described as following:

$$\begin{aligned} L(W_{i})= & {} Tr(\Psi _{i-1}^T\Psi _{i-1}W_{i}\Phi _{i+1}V_{i}V_{i}^T\Phi _{i+1}^TW_{i}^T\nonumber \\{} & {} +\Phi _{i+1}\Phi _{i+1}^TW_{i}^T\Psi _{i-1}^T X X^T\Psi _{i-1} W_{i})\nonumber \\{} & {} -4Tr(\Phi _{i+1}V_{i}X^T \Psi _{i-1}W_{i}) \end{aligned}$$

(A5)

Let

$$\begin{aligned} \begin{aligned} G((W_{i}),(W_{i})')=&\sum _{kl}\frac{(\Psi _{i-1}^T\Psi _{i-1} W_{i}'\Phi _{i+1} V_{i} V_{i}^T \Phi _{i+1}^T)_{kl}((W_{i})_{kl})^2}{(W_{i})'_{kl}}\\ +&\sum _{kl}\frac{(\Psi _{i-1}^TXX^T\Psi _{i-1}W_{i}'\Phi _{i+1} \Phi _{i+1}^T)_{kl} ((W_{i})_{kl})^2}{(W_{i})'_{kl}}\\ -&4(\Psi _{i-1}^TX V_{i}^T\Phi _{i+1}^T)_{kl}(W_{i})'_{ij}(1+\log {\frac{(W_{i})_{kl}}{(W_{i})'_{kl}}}) \end{aligned} \end{aligned}$$

(A6)

be an auxiliary function of $L(W_{i})$. Obviously, when $(W_{i})=(W_{i})'$, $G((W_{i}),(W_{i})')=L(W_{i})$. Therefore, we need to prove $G((W_{i}),(W_{i})')\le L(W_{i})$ here. When $(W_{i})')\ne L(W_{i})$, according to Lemma 3.3 and inequation $z\ge 1+\log z$, we get the following inequations:

$$\begin{aligned}{} & {} \sum _{kl}\frac{(\Psi _{i-1}^T\Psi _{i-1} W_{i}'\Phi _{i+1} V_{i} V_{i}^T \Phi _{i+1}^T)_{kl}((W_{i})_{kl})^2}{(W_{i})'_{kl}}\nonumber \\{} & {} \quad>Tr(\Psi _{i-1}^T\Psi _{i-1}W_{i}\Phi _{i+1}V_{i}V_{i}^T\Phi _{i+1}^TW_{i}^T) \nonumber \\{} & {} \sum _{kl}\frac{(\Psi _{i-1}^TXX^T\Psi _{i-1}W_{i}'\Phi _{i+1} \Phi _{i+1}^T)_{kl} ((W_{i})_{kl})^2}{(W_{i})'_{kl}} \nonumber \\{} & {} \quad>Tr(\Phi _{i+1}\Phi _{i+1}^TW_{i}^T\Psi _{i-1}^T X X^T\Psi _{i-1} W_{i}) \nonumber \\{} & {} Tr(\Phi _{i+1}V_{i}X^T \Psi _{i-1}W_{i})>\sum _{kl}(\Psi _{i-1}^TX V_{i}^T\Phi _{i+1}^T)_{kl}(W_{i})'_{ij}\nonumber \\{} & {} \quad \times (1+\log {\frac{(W_{i})_{kl}}{(W_{i})'_{kl}}}) \end{aligned}$$

(A7)

Summing all above bounds and we get the following inequations:

$$\begin{aligned} G((W_{i}),(W_{i})')>L(W_{i}) \end{aligned}$$

(A8)

Hence, $G((W_{i}),(W_{i})')$ is an auxiliary function of $L(W_{i})$. To get the global minimum value of function $G((W_{i}),(W_{i})')$, we fix the $W_{i}'$ and take the derivative of $G((W_{i}),(W_{i})')$ on $(W_{i})_{kl}$.

$$\begin{aligned} \begin{aligned} \frac{\partial G((W_{i}),(W_{i})')}{\partial ( W_{i})_{kl})}=&2\frac{(\Psi _{i-1}^T\Psi _{i-1} W_{i}'\Phi _{i+1} V_{i} V_{i}^T \Phi _{i+1}^T)_{kl}((W_{i})_{kl})}{(W_{i})'_{kl}}\\ +&2\frac{(\Psi _{i-1}^TXX^T\Psi _{i-1}W_{i}'\Phi _{i+1} \Phi _{i+1}^T)_{kl} ((W_{i})_{kl})}{(W_{i})'_{kl}}\\ -&4(\Psi _{i-1}^TX V_{i}^T\Phi _{i+1}^T)_{kl}(\frac{(W_{i})'_{kl}}{(W_{i})_{kl}}) \end{aligned} \end{aligned}$$

(A9)

Because the Hessian matrix $\frac{\partial ^{2} G((W_{i}),(W_{i})_{kl}')}{\partial (W_{i})_{kl}\partial ( W_{i})_{kl}}$ is positive define, the $G((W_{i}),(W_{i})')$ is a convex function. Setting $\frac{\partial G((W_{i}),(W_{i})_{kl}')}{\partial ( W_{i})_{kl})}$=0 and based on the KTT conditions, we get the following updating rules:

$$\begin{aligned} W_{i}\!\leftarrow \! W_{i}'\odot \frac{2\Psi _{i-1}^T XV_{p}^T\Phi _{i+1}^T}{\Psi _{i-1}^T\Psi _{i-1}W_{i}'\Phi _{i+1}^T V_{p} V_{p}^T \Phi _{i+1}^T\!+\!\Psi _{i-1}^T X X^T \Psi _{i-1}W_{i}'\Phi _{i+1}\Phi _{i+1}^T} \end{aligned}$$

(A10)

Noting $W^{t+1}\leftarrow W$ and $W^{t}\leftarrow W'$, the (A10) recovers the (17).Thus, the objective function L monotonically decreases according to the updating rule. Equation (17) for L.

Next, we will prove the theorem 2. Let us rewrite the (18) here:

$$\begin{aligned} L(V_{i})=||X-\Psi _{i}V_{i}||^2_{F}+||V_{i}-\Psi _{i}^T X||_{F}^2+\beta Tr(V_{i}^T D V_{i}) \end{aligned}$$

(A11)

According to the relationship between matrix norm and matrix trace, we rewrite the equation L with removing irrelevant elements:

$$\begin{aligned} L(V_{i})=Tr(-4X^T\Psi _{i} V_{i}+V_{i}^TV_{i}+V_{i}^TDV_{i}+V_{i}^T\Psi _{i}^T\Psi _{i} V_{i}) \end{aligned}$$

(A12)

Now an auxiliary function of (A12) is

$$\begin{aligned} \begin{aligned} G((V_{i}),(V_{i})')&\!=\!\sum _{kl}\frac{(\Psi _{i-1}^T\Psi _{i} V_{i}')_{kl}(V_{i})^2_{kl}}{(V_{i}')_{kl}}\!+\!\sum _{kl}(V_{i})_{kl}(V_{i})_{kl}\\ {}&+\sum _{kl}(V_{i})_{kl}D(V_{i})_{kl}-4\sum _{kl}(\Psi _{i}^T X)_{kl}(V_{i})'_{kl}\\&\times (1+\log \frac{(V_{i})_{kl}}{(V_{i})'_{kl}}) \end{aligned} \end{aligned}$$

(A13)

When $(V_{i})'=(V_{i})$, clearly $G(V_{i},V_{i}')=L(V_{i})$. Hence, we need to prove $G(V_{i},V_{i}')>L(V_{i})$ here. Based on Lemma 3.3 and $z\ge 1+\log z$, we get the following inequations:

$$\begin{aligned} \begin{aligned} \sum _{kl}\frac{(\Psi _{i-1}^T\Psi _{i} V_{i}')_{kl}(V_{i})^2_{kl}}{(V_{i}')_{kl}}>Tr(V_{i}^T\Psi _{i}^T\Psi _{i} V_{i})\\ Tr(X^T\Psi _{i}V_{i})>\sum _{kl}(\Psi _{i}^T X)_{kl}(V_{i})'_{kl}(1+\log \frac{(V_{i})_{kl}}{(V_{i})'_{kl}}) \end{aligned} \end{aligned}$$

(A14)

The first term of (A13) is always bigger than the fourth term of (A12) and the fourth term of (A13) is always smaller than the first term of (A12). Therefore, we get $G((V_{i}),(V_{i})')>L(V_{i})$. Our objective now is to obtain the global minimum value of $G((V_{i}),(V_{i}'))$. The derivative of $G((V_{i}),(V_{i})')$ is

$$\begin{aligned} \begin{aligned} \frac{\partial G((V_{i}),(V_{i})')}{\partial (V_{i})_{kl}}&=2\frac{(\Psi _{i-1}^T\Psi _{i} V_{i}')_{kl}(V_{i})_{kl}}{(V_{i})'_{kl}}+2(V_{i})_{kl}\\ {}&+2D(V_{i})_{kl}-4(\Psi _{i}^TX)_{kl}\frac{(V_{i})'_{kl}}{(V_{i})_{kl}} \end{aligned} \end{aligned}$$

(A15)

Because the Hessian matrix $\frac{\partial ^{2} G((V_{i}),(V_{i})_{kl}')}{\partial (V_{i})_{kl}\partial ( V_{i})_{kl}}$ is positive define and the $G((V_{i}),(V_{i})')$ is a convex function. Setting $\frac{\partial G((V_{i}),(V_{i})_{kl}')}{\partial ( V_{i})_{kl})}$=0 and according to the KTT condition, we get the following updating rules:

$$\begin{aligned} V_{i}\leftarrow V_{i}'\odot \frac{2\Psi _{i}X}{\Psi _{i-1}^T\Psi _{i-1}V_{i}'+V_{i}'+\beta DV_{i}'} \end{aligned}$$

(A16)

Noting $V^{t+1}\leftarrow V$ and $V^{t}\leftarrow V'$, the (A16) recovers the (22). Therefore, objective function L monotonically decreases under the update rule (22) for L.