Deep manifold matrix factorization autoencoder using global connectivity for link prediction

Abstract

Link prediction aims to predict missing links or eliminate spurious links by employing known complex network information. As an unsupervised linear feature representation method, matrix factorization (MF)-based autoencoder (AE) can project the high-dimensional data matrix into the low-dimensional latent space. However, most of the traditional link prediction methods based on MF or AE adopt shallow models and single adjacency matrices, which cannot adequately learn and represent network features and are susceptible to noise. In addition, because some methods require the input of symmetric data matrix, they can only be used in undirected networks. Therefore, we propose a deep manifold matrix factorization autoencoder model using global connectivity matrix, called DM-MFAE-G. The model utilizes PageRank algorithm to get the global connectivity matrix between nodes for the complex network. DM-MFAE-G performs deep matrix factorization on the local adjacency matrix and global connectivity matrix, respectively, to obtain global and local multi-layer feature representations, which contains the rich structural information. In this paper, the model is solved by alternating iterative optimization method, and the convergence of the algorithm is proved. Comprehensive experiments on different real networks demonstrate that the global connectivity matrix and manifold constraints introduced by DM-MFAE-G significantly improve the link prediction performance on directed and undirected networks.

Appendix

1.1 Convergence proof of DM-MFAE-G

Here, we demonstrate the convergence of DM-MFAE-G, i.e., $\mathcal {L}$ is non-increasing if under the update rules (27), (28) and (29). We first prove the following Theorems 1 to 3.

Theorem 1

Updating $U_{\textrm{i}}$ using the rule of (27) while fixing $W_{\textrm{i}}$ and $H_{\textrm{i}}$, the object function of $\mathcal {L}$ monotonically decreases.

Theorem 2

Updating $W_{\textrm{i}}$ using the rule of (28) while fixing $U_{\textrm{i}}$ and $H_{\textrm{i}}$, the object function of $\mathcal {L}$ monotonically decreases.

Theorem 3

Updating $H_{\textrm{i}}$ using the rule of (29) while fixing $U_{\textrm{i}}$ and $W_{\textrm{i}}$, the object function of $\mathcal {L}$ monotonically decreases.

Lemma 1

Under the rule of (27), the following inequation holds

$$\begin{aligned}&\varphi (X,{W_i},U_i^{t + 1}) + \varphi (C,{H_i},U_i^{t + 1}) + \beta {\left\| {U_i^{t + 1}} \right\| _{2,1}} \nonumber \\&\le \varphi (X,{W_i},U_i^t) + \varphi (C,{H_i},U_i^t) + \beta {\left\| {U_i^t} \right\| _{2,1}} , \end{aligned}$$

(33)

where

$$\begin{aligned} \varphi \left( \! A,B,U_i \right)= & {} Tr\left( -4A^T\psi _{i-1}U_iB\!+\!B^TU_{i}^{T}\psi _{i-1}^{T}\psi _{i-1}U_i{B}\right. \\{} & {} \left. +A^T\psi _{i-1}U_iU_{i}^{T}\psi _{i-1}^{T}A \right) , \end{aligned}$$

t is the iteration number of times.

Proof

We propose lemmas and definitions based on D. Lee et al. [44] to aid us prove Lemma 1.$\square$

Definition 1

Let function $G(h,h^{'})$ is an auxiliary function of the function L(h), if $\textrm{ }G\left( H,H^{\mathrm {'}} \right) \ge L\left( h \right) ,G\left( h,h^{\mathrm {'}} \right) =L\left( h \right)$, for any $h,h^{\mathrm {'}}$ is satisfied.

Lemma 2

If $G(h,h^{\mathrm {'}})$ is an auxiliary function for L(h), then L(h) is non-increasing under the update

$$\begin{aligned} {h^{t + 1}} = \arg \mathop {\min }\limits _h G(h,h'). \end{aligned}$$

(34)

Proof

$L\left( h^{t+1} \right) \le G\left( h^{t+1} \right) ,h^t\le G\left( h^t \right) ,h^t=L\left( h^t \right)$.$\square$

Next, we rewrite the object function $\mathcal {L}$ and remove the irrelevant terms to get a function that is only related to $U_i$.

$$\begin{aligned} \mathcal{L}({U_i})= & {} Tr( - 4{X^T}\psi _{i - 1}^{}{U_i}{W_i} + W_i^TU_i^T\psi _{i - 1}^T{\psi _{i - 1}}{U_i}{W_i})\nonumber \\{} & {} + Tr(X_{}^T{\psi _{i - 1}}{U_i}U_i^T\psi _{i - 1}^T{X_{}})\nonumber \\{} & {} + \alpha Tr( - 4{C^T}\psi _{i - 1}^{}{U_i}{H_i}\nonumber \\{} & {} + H_i^TU_i^T\psi _{i - 1}^T{\psi _{i - 1}}{U_i}{H_i} )\nonumber \\{} & {} + Tr(C_{}^T{\psi _{i - 1}}{U_i}U_i^T\psi _{i - 1}^TC)+\beta {\left\| {{U_i}} \right\| _{2,1}}. \end{aligned}$$

(35)

Then we prove that the (27) is updated according to the rules of (34) under a certain auxiliary function. Considering that for the element $\left( U_i \right) _{kl}$ of $U_i$. The part of the objective function only related to $\left( U_i \right) _{kl}$ is denoted as $\mathcal {L} _{\left( U_i \right) _{kl}}$, the derivative of $\left( U_i \right) _{kl}$ can be obtained:

$$\begin{aligned} {{\mathcal{L}'}_{({U_i})kl}}= & {} ( - {4^{}}\psi _{i - 1}^T{X_i}W_i^T + 2{A_1}{U_i}{A_2} + 2{U_i}{A_4}) \nonumber \\{} & {} + \alpha ( - 4\psi _{i - 1}^TCH_i^T + 2{A_1}{U_i}{A_3} + 2{U_i}{A_5}) \nonumber \\{} & {} + 2\beta {M_1}{U_i}{)_{kl}}. \end{aligned}$$

(36)

$$\begin{aligned} {{\mathcal{L}''}_{({U_i})kl}}= & {} 2{({A_1})_{kk}}{({A_2})_{ll}} + 2\alpha {({A_1})_{kk}}{({A_3})_{ll}}\nonumber \\{} & {} + 2{({A_4} + \alpha {A_5})_{ll}} + 2\beta {({M_1})_{kk}} \end{aligned}$$

(37)

where $A_1=\psi _{i-1}^{T}\psi _{i-1}$, $A_2=W_iW_{i}^{T}$, $A_3=H_iH_{i}^{T}$,$\textrm{ }A_4=\psi _{i-1}^{T}XX^T\psi _{i-1}$, $A_5=\psi _{i-1}^{T}CC^T\psi _{i-1}$. Therefore, it is shown that for each $\left( U_i \right) _{kl}$, it is updated according to the rules of (34).

Lemma 3

Assuming that $\mathcal {L} ^{\mathrm {'}}$ represents the first derivative with respect to the variable $U_i$, the Eq. (38) is an auxiliary function of $\mathcal {L} _{\left( U_i \right) _{kl}}$.

$$\begin{aligned} {\begin{matrix} &{}G({U_i},({U_i})_{kl}^t)={\mathcal{L}_{({U_i})kl}}(({U_i})_{kl}^t) + {{\mathcal{L}'}_{({U_i})kl}}({U_i} - ({U_i})_{kl}^t) \\ {} &{}+ \frac{{{{({A_1}{U_i}{A_2} + \alpha {A_1}{U_i}{A_3} + {U_i}{A_4} + \alpha {U_i}{A_5} + \beta {M_1}{U_i})}_{kl}}}}{{({U_i})_{kl}^t}}{({U_i} - ({U_i})_{kl}^t)^2}. \end{matrix}} \end{aligned}$$

(38)

Proof

Clearly, $G\left( U_i,U_i \right) =\mathcal {L} _{\left( U_i \right) _{kl}}\left( U_i \right)$

We will prove that $G\left( U_i,\left( U_i \right) _{kl}^{t} \right) \ge \mathcal {L} _{\left( U_i \right) _{kl}}\left( U_i \right)$.

Firstly, the Taylor expansion of $\mathcal {L} _{\left( U_i \right) _{kl}}\left( U_i \right)$ is as follows:

$$\begin{aligned} {\begin{matrix} {\mathcal{L}_{({U_i})kl}}({U_i})=&{}{\mathcal{L}_{({U_i})kl}}(({U_i})_{kl}^t) + {{\mathcal{L}'}_{({U_i})kl}}({U_i} - ({U_i})_{kl}^t)\\ &{}+ \frac{1}{2}{{\mathcal{L}''}_{({U_i})kl}}{({U_i} - ({U_i})_{kl}^t)^2}, \end{matrix}} \end{aligned}$$

(39)

Then, by comparing (38) and (39), it can be found that to make $G\left( U_i,\left( U_i \right) _{kl}^{t} \right) \ge \mathcal {L} _{\left( U_i \right) _{kl}}\left( U_i \right)$ hold, the following inequality necessary to be satisfied

$$\begin{aligned} {\begin{matrix} &{}{({A_1}{U_i}{A_2} + \alpha {A_1}{U_i}{A_3} + {U_i}{A_4} + \alpha {U_i}{A_5} + \beta {M_1}{U_i})_{kl}}\\ &{}\ge (U_i^T)_{kl}^t[{({A_1})_{kk}}{({A_2})_{ll}} + \alpha {({A_1})_{kk}}{({A_3})_{ll}} + {({A_4} + \alpha {A_5})_{ll}}\\ &{} + \beta {({M_1})_{kk}}] . \end{matrix}} \end{aligned}$$

(40)

Comparing both sides of the inequality, it is clearly that (41)   (44) hold:

$$\begin{aligned} {({A_1}{U_i}{A_2})_{kl}}= & {} \sum \limits _a {\sum \limits _b {{{({A_1})}_{ka}}({U_i})_{ab}} } {({A_2})_{bl}}\nonumber \\\ge & {} ({U_i})_{kl}^t{({A_1})_{kk}}({A_2})_{ll}, \end{aligned}$$

(41)

$$\begin{aligned} {(\alpha {A_1}{U_i}{A_3})_{kl}}= & {} \alpha \sum \limits _a {\sum \limits _b {{{({A_1})}_{ka}}({U_i})_{ab}} } {({A_3})_{bl}}\nonumber \\\ge & {} \alpha ({U_i})_{kl}^t{({A_1})_{kk}}{({A_3})_{ll}} ,\end{aligned}$$

(42)

$$\begin{aligned} {({U_i}{A_4} + \alpha {U_i}{A_5})_{kl}}= & {} \sum \limits _a {({U_i})_{ka}{{({A_4} + \alpha {A_5})}_{al}}}\nonumber \\\ge & {} ({U_i})_{kl}^t{({A_4} + \alpha {A_5})_{ll}} ,\end{aligned}$$

(43)

$$\begin{aligned} {(\beta {M_1}{U_i})_{kl}}= & {} \beta \sum \limits _a {{{({A_4} + \alpha {A_5})}_{ka}}({U_i})_{al}} \nonumber \\\ge & {} \beta ({U_i})_{kl}^t{({M_1})_{kk}}. \end{aligned}$$

(44)

Thus, inequality (40) holds. $G\left( U_i,\left( U_i \right) _{kl}^{t} \right)$ is an auxiliary function of $\mathcal {L} _{\left( U_i \right) _{kl}}$.$\square$

Based on the properties of auxiliary functions, Lemma 1 can be proved.

To find the globe minimum of $G\left( U_i,\left( U_i \right) _{kl}^{t} \right)$ with $U_{\textrm{i}}$ fixed, we take the derivative of $G\left( U_i,\left( U_i \right) _{kl}^{t} \right)$ on $U_{\textrm{i}}$, then according to KKT conditions we get:

$$\begin{aligned}{} & {} \frac{{\partial G({U_i},({U_i})_{kl}^t)}}{{\partial {U_i}}}\\= & {} {L }_{({U_i})kl}^{'}(({U_i})_{kl}^t) \\{} & {} + \frac{{2{{({A_1}{U_i}{A_2} \!+\! \alpha {A_1}{U_i}{A_3} \!+\! {U_i}{A_4} \!+\! \alpha {U_i}{A_5} + \beta {M_1}{U_i})}_{kl}}}}{{({U_i})_{kl}^t}}\\{} & {} \times ({U_i} - ({U_i})_{kl}^t)\\= & {} {L }_{({U_i})kl}^{'}(({U_i})_{kl}^t) \\{} & {} \!\!\!\!+ \frac{{2{{({A_1}{U_i}{A_2} \!+\! \alpha {A_1}{U_i}{A_3}\! +\! {U_i}{A_4} \!+\! \alpha {U_i}{A_5} \!+\! \beta {M_1}{U_i})}_{kl}}}}{{({U_i})_{kl}^t}}{U_i}\\{} & {} \!\!\!\!- 2{({A_1}{U_i}{A_2} + \alpha {A_1}{U_i}{A_3} + {U_i}{A_4} + \alpha {U_i}{A_5} + \beta {M_1}{U_i})_{kl}}\\= & {} 0. \end{aligned}$$

According to (36), we have

$$\begin{aligned} {({U_i})_{kl}} \leftarrow ({U_i})_{kl}^t\frac{{{{(2\psi _{i - 1}^TXV_i^T + 2\alpha \psi _{i - 1}^TCH_i^T)}_{kl}}}}{{{{({Y_i})}_{kl}}}}. \end{aligned}$$

(45)

Therefore, the iteration rule obtained by the auxiliary function (38) under the update (34) is exactly the iteration (27) for $U_i$. Since the Hessian matrix $\frac{\partial ^2G\left( U_i,\left( U_i \right) _{kl}^{t} \right) }{\partial \left( U_i \right) _{kl}\partial \left( U_i \right) _{kl}}$ is positive define and $G\left( U_i,\left( U_i \right) _{kl}^{t} \right)$ is a convex function. Thus, objective function $\mathcal {L}$ is non-increasing under the update rule (38) for $\mathcal {L}$, and Theorem 1 is proved.

Due to limited space, the proof of Theorem 2 and 3 are omitted.Because there is a lower bound because the loss function $\mathcal {L}>0$ holds constant. Therefore, the DM-MFAE-G model converges.

1.2 More details about the experiments

We report the deep factorized feature dimension of each layer $R=[r_1,r_2,...,r_p]$ in Table 7.

Table 7 The parameters $R$ in ten datasets

Meanwhile, we consider that the dataset needs to be randomly divided into a training set and a test set first in link prediction experiments, the optimal parameters of the model will change with the different divisions because the division of data is random. Therefore, we give below the approximate range of the optimal parameters $\alpha$, $\beta$, $\gamma$ appearing in each data set during the missing link prediction experiments as a reference in Table 8.

Table 8 Optimal parameter range in missing link prediction experiments