Incomplete Multi-view Learning via Consensus Graph Completion

Abstract

Traditional graph-based multi-view learning methods usually assume that data are complete. Whereas several instances of some views may be missing, making the corresponding graphs incomplete and reducing the virtue of graph regularization. To mitigate the negative effect, a novel method, called incomplete multi-view learning via consensus graph completion (IMLCGC), is proposed in this paper, which completes the incomplete graphs based on the consensus among different views and then fuses the completed graphs into a common graph. Specifically, IMLCGC develops a learning framework for incomplete multi-view data, which contains three components, i.e., consensus low-dimensional representation, graph regularization, and consensus graph completion. Furthermore, a generalization error bound of the model is established based on Rademacher’s complexity. It shows the theory that learning with incomplete multi-view data is difficult. Experimental results on six well-known datasets indicate that IMLCGC significantly outperforms the state-of-the-art methods.

Appendix A

1.1 A.1 Proof of Lemma 1

For convenience, let’s denote

$$\begin{aligned} g\left( D \right) =\mu {{\left\| D \right\| }_{*}}+\gamma \left\| D \right\| _{F}^{2}+\lambda tr\left( {{F}^{T}}D \right) . \end{aligned}$$

Before giving proof of Lemma 1, we first give the Lemma 3.

Lemma 3

Let $Y\in \partial g\left( D \right)$ and $Y'\in \partial g\left( D' \right)$, then the following inequality is hold

$$\begin{aligned} \left\langle Y-{Y}',D-{D}' \right\rangle \ge 2\gamma \left\| D-{D}' \right\| _{F}^{2}. \end{aligned}$$

(A1)

Proof

According to $Y\in \partial g\left( D \right)$, we have $Y=\mu {{Y}_{0}}+2\gamma D+\lambda F$, and similarly ${Y}'=\mu {Y'_0}+2\gamma {D}'+\lambda F$, where $Y_0\in \partial \left\| D \right\| _*$. There is

$$\begin{aligned} \left\langle Y-Y',D-D' \right\rangle =\mu \left\langle {Y_0}-Y'_0,D-D' \right\rangle +2\gamma \left\| D-{D}' \right\| _{F}^{2}. \end{aligned}$$

Let’s prove $\left\langle {{Y}_{0}}-{Y'_0},D-{D}' \right\rangle >0$. From the definition of $Y_0$, we can get ${{\left\| {{Y}_{0}} \right\| }_{2}}\le 1$ and $\left\langle {{Y}_{0}},D \right\rangle ={{\left\| D \right\| }_{*}}$, then [34, 45]

$$\begin{aligned} \left\langle {{Y}_{0}},{D}' \right\rangle \le {{\left\| {{Y}_{0}} \right\| }_{2}}{{\left\| {{D}'} \right\| }_{*}}\le {{\left\| {{D}'} \right\| }_{*}}, \end{aligned}$$

therefore

$$\begin{aligned} \left\langle {{Y}_{0}}-{Y'_0},D-{D}' \right\rangle ={{\left\| D \right\| }_{*}}\text {+}{{\left\| {{D}'} \right\| }_{*}}-\left\langle {Y'_0},D \right\rangle -\left\langle {{Y}_{0}},{D}' \right\rangle \ge 0. \end{aligned}$$

So, Eq. (A1) holds. $\square$

The proof of the Lemma 1 is given as follows.

Proof

Let $D^*$ and $\Lambda ^*$ are the primal and dual optimal solution of the problem (11). Then the optimality condition is

$$\begin{aligned} {{Y}^{k}}={{P}_{\Omega }}\left( {{\Lambda }^{k-1}} \right) ,\quad {{Y}^{*}}={{P}_{\Omega }}\left( {{\Lambda }^{*}} \right) , \end{aligned}$$

where $Y^*\in \partial g\left( D^* \right)$, $Y^k\in \partial g\left( D^k \right)$ for any k. Then we can get

$$\begin{aligned} {{Y}^{k}}-{{Y}^{*}}={{P}_{\Omega }}\left( {{\Lambda }^{k-1}}-{{\Lambda }^{*}} \right) \end{aligned}$$

and further according to Lemma 3 to obtain

$$\begin{aligned} \left\langle {{P}_{\Omega }}\left( {{\Lambda }^{k-1}}-{{\Lambda }^{*}} \right) ,{{D}^{k}}-{{D}^{*}} \right\rangle =\left\langle {{Y}^{k}}-{{Y}^{*}},{{D}^{k}}-{{D}^{*}} \right\rangle \ge 2\gamma \left\| {{D}^{k}}-{{D}^{*}} \right\| _{F}^{2}. \end{aligned}$$

(A2)

From the ${{P}_{\Omega }}\left( {{D}^{*}} \right) ={{P}_{\Omega }}\left( M \right)$ we have

$$\begin{aligned} {{\left\| {{P}_{\Omega }}\left( {{\Lambda }^{k}}-{{\Lambda }^{*}} \right) \right\| }_{F}}={{\left\| {{P}_{\Omega }}\left( {{\Lambda }^{k-1}}-{{\Lambda }^{*}} \right) +\rho {{P}_{\Omega }}\left( {{D}^{*}}-{{D}^{k}} \right) \right\| }_{F}}. \end{aligned}$$

(A3)

Let ${{r}_{k}}={{\left\| {{P}_{\Omega }}\left( {{\Lambda }^{k}}-{{\Lambda }^{*}} \right) \right\| }_{F}}$, then the following formula holds according to Eqs. (A2) and (A3),

$$\begin{aligned} \begin{aligned} r_{k}^{2}&=r_{k-1}^{2}-2\rho \left\langle {{P}_{\Omega }}\left( {{\Lambda }^{k-1}}-{{\Lambda }^{*}} \right) ,{{D}^{k}}-{{D}^{*}} \right\rangle +{{\rho }^{2}}\left\| {{P}_{\Omega }}\left( {{D}^{k}}-{{D}^{*}} \right) \right\| _{F}^{2} \\&\le r_{k-1}^{2}-\left( 4\gamma \rho -{{\rho }^{2}} \right) \left\| {{P}_{\Omega }}\left( {{D}^{k}}-{{D}^{*}} \right) \right\| _{F}^{2} . \end{aligned} \end{aligned}$$

Because $0<\rho <4\gamma$, so $4\gamma \rho -{{\rho }^{2}}>0$, which further has the following two properties:

• 1. The sequence $\left\{ {{\left\| {{P}_{\Omega }}\left( {{\Lambda }^{k}}-{{\Lambda }^{*}} \right) \right\| }_{F}}\right\}$ is nonincreasing, and is convergent due to it has a lower bound.

• 2. Therefore, $\left\| {{P}_{\Omega }}\left( {{D}^{k}}-{{D}^{*}} \right) \right\| _{F}^{2}\rightarrow 0$ with $k\rightarrow \infty$.

$\square$

1.2 A.2 Proof of Theorem 1

According to the above analysis, the optimal solution can be obtained for both of two subproblems. Therefore, Algorithm 1 makes the objective function value of Eq. (7) decrease monotonically, and because it has a lower bound, the convergence is guaranteed. Assume that Algorithm 1 converges to $A^*$, $\{W_i^*\}_{i=1}^m$, and $D^*$, and then prove that it is KKT point.

Proof

The Lagrangian function of problem (7) is

$$\begin{aligned} \mathcal {L}_2\left( A,\{W_i\}_{i=1}^m,D,\Gamma ,\Lambda \right) =\sum \limits _{i=1}^{m}{\left\| \left( A-W_i^TX_i\right) P_i\right\| _F^2}+\left\langle \Gamma ,\left( AA^T-I\right) \right\rangle +\mathcal {L}_1\left( D,\Lambda \right) , \end{aligned}$$

(A4)

where $\Gamma$ is Lagrange multiplier. Taking the derivative w.r.t. A, $\{W_i\}_{i=1}^m$, D, $\Gamma$, and $\Lambda$ respectively and setting them to zero, we can get the KKT condition of the problem (7):

According to the solving step of A and $\{W_i\}_{i=1}^m$, $A^*$ and $\{W_i^*\}_{i=1}^m$ satisfy the following equation,

$$\begin{aligned} W_i^*=\left( X_iP_iX_i^T\right) ^{-1}X_iP_i{A^*}^T \qquad i=1,2,\cdots ,m. \end{aligned}$$

(A6)

Further, $A^*$ is obtained by solving the eigenvalue problem of Eq. (10), so Eqs. (A5a), (A5b) and (A5d) is established. Since the essence of SVT is to solve D through Eq. (A5c) , therefore Eq. (A5c) obviously is satisfied. We know from Lemma 1 that $\left\| {\mathcal {P}_{\Omega }}\left( {{D}^{k}}-{{D}^{*}} \right) \right\| _{F}^{2}\rightarrow 0$ with $k\rightarrow \infty$, so Eq. (A5e) is satisfied. In summary, Algorithm 1 will converge to KKT point of problem (7). $\square$

1.3 A.3 Proof of Lemma 2

Proof

From the definition of $p_i\left( x\right)$ and algebraic operation, we get formulation as follows,

$$\begin{aligned} \begin{aligned} f(x)&=\sum \limits _{i=1}^{m}{\left\| (a-W_i^Tx_i)p_i\left( x\right) \right\| _2^2} \\&=\sum \limits _{i=1}^{m}{\left[ p_i\left( x\right) a^Ta-2p_i\left( x\right) a^TW_i^Tx_i+p_i\left( x\right) x_i^TW_iW_i^Tx_i \right] } \\&=\sum \limits _{i=1}^{m}{p_i\left( x\right) \left[ a^Ta-2\left( {{a}^{T}}{{W}_i}^T\right) {{x}_i} +\text {vec}{{\left( {{W}_i}W^{T}_i \right) }^{T}}\text {vec}\left( {{x}_i}{{x}_i}^{T} \right) \right] }. \end{aligned} \end{aligned}$$

Let

$$\begin{aligned} \begin{aligned} w&={{\left[ {{w}_{1}}^{T},{{w}_{2}}^{T},\cdots ,{{w}_{m}}^{T} \right] }^{T}}, \\ {{\Phi }_{p}}\left( {{x}} \right)&={{\left[ p_{1}\left( x\right) \varphi {{\left( x_{1} \right) }^{T}},p_{2}\left( x\right) \varphi {{\left( x_{2} \right) }^{T}},\cdots , p_{m}\left( x\right) \varphi {{\left( x_{m} \right) }^{T}} \right] }^{T}}, \\ \end{aligned} \end{aligned}$$

where $w_i=\left[ a^T,-2a^TW_i^T,\text {vec}\left( W_iW_i^T\right) ^T \right] ^T$ and $\varphi \left( x_i\right) =\left[ a^T,x_i^T,\text {vec}\left( x_ix_i^T\right) ^T \right] ^T$ for $i=1, 2, \dots , m$. Therefore we can rewrite Eq. (18) as follows,

$$\begin{aligned} f(x)=\left\langle w,\Phi _{p} \left( x \right) \right\rangle . \end{aligned}$$

(A7)

So it is easy to see that f(x) is a linear function for $\Phi _{p} \left( x \right)$. It is shown below that the feature space $\mathcal {F}$ is derived from $\hat{k}_p\left( x,y\right)$,

$$\begin{aligned} \begin{aligned} \hat{k}_p\left( x,y\right)&=\sum \limits _{i=1}^{m}{p_i\left( x\right) p_i\left( y\right) \left[ a^Ta+k\left( x_i,y_i\right) +k\left( x_i,y_i\right) ^2\right] } \\&=\sum \limits _{i=1}^{m}{p_i\left( x\right) p_i\left( y\right) \left[ a^Ta+{x_i}^Ty_i+{x_i}^Ty_i{x_i}^Ty_i\right] } \\&=\left\langle \Phi _p \left( x \right) ,\Phi _p \left( y \right) \right\rangle . \end{aligned} \end{aligned}$$

$\square$

1.4 A.4 Proof of Theorem 2

Proof

Let’s first derive the upper bound on $\left\| w\right\| _2^2$ as follows,

$$\begin{aligned} \begin{aligned} \left\| w\right\| _2^2&=\sum \limits _{i=1}^{m}{\left\| w_i\right\| _2^2} \\&=\sum \limits _{i=1}^{m}{\left[ a^Ta+4a^TW_i^TW_ia+\text {vec}\left( W_i^TW_i\right) ^T\text {vec}\left( W_iW_i^T\right) \right] } \\&=\sum \limits _{i=1}^{m}{\left[ a^Ta+4\text {vec}\left( W_i^TW_i\right) ^T\text {vec}\left( aa^T\right) +\text {vec}\left( W_iW_i^T\right) ^T\text {vec}\left( W_iW_i^T\right) \right] } \\&\le m\left( c_1^2+4c_1^2c_2+c_2^2\right) . \end{aligned} \end{aligned}$$

So $\left\| w\right\| _2<B$. Based on Lemma 2 and assumptions, it is easy to find that $f\left( x\right)$ belongs to the function class

$$\begin{aligned} \mathcal {F}_B=\left\{ x\rightarrow \left\langle w,\Phi _p\left( x\right) \right\rangle :\left\| w\right\| _2\le B\right\} . \end{aligned}$$

Obviously, $f\left( x\right) \ge 0$, and we are easy going to show that $f\left( x\right)$ has an upper bound

$$\begin{aligned} \begin{aligned} f\left( x\right)&=\left\langle w,\Phi _p\left( x\right) \right\rangle \le \left\| w\right\| _2\left\| \Phi _p\left( x\right) \right\| _2\\&=B\sqrt{\left\langle \Phi _p\left( x\right) ,\Phi _p\left( x \right) \right\rangle }=B\sqrt{\hat{k}_p\left( x,x\right) } \\&\le BR. \end{aligned} \end{aligned}$$

As a result, we did this by exploiting the McDiarmid’s concentration inequality [27, 46], the follows inequation holds with probability at lest $1-\delta$ that

$$\begin{aligned} \mathbb {E}\left[ f\left( x \right) \right] \le \frac{1}{n}\sum \limits _{k=1}^{n}{f\left( {{x}_{k}} \right) }+2\hat{R}_n\left( \mathcal {F}_B \right) +3BR\sqrt{\frac{\ln \left( {2}/{\delta }\; \right) }{2n}}, \end{aligned}$$

(A8)

where $\hat{R}_n\left( \mathcal {F}_{B}\right)$ is empirical Rademacher’s complexity of $\mathcal {F}_{B}$. Now let’s estimate the upper bound of $\hat{R}_n\left( \mathcal {F}_{B}\right)$. According to the definition of $\hat{R}_n\left( \mathcal {F}_{B}\right)$, we have following inequation holds,

$$\begin{aligned} \begin{aligned} \hat{R}_n\left( \mathcal {F}_{B}\right)&=\frac{1}{n}\mathbb {E}_\sigma \left[ \underset{f\in \mathcal {F}_{B}}{\mathop {\sup }}\sum \limits _{k=1}^{n}{\sigma _kf\left( x^{\left( k\right) }\right) }\right] \\&=\frac{1}{n}{\mathbb {E}_{\sigma }}\left\{ \underset{w,p}{\mathop {\sup }}\,\sum \limits _{k=1}^{n}{\sigma _k\left\langle w,\Phi _p\left( x^{\left( k\right) }\right) \right\rangle } \right\} \\&=\frac{1}{n}\mathbb {E}_\sigma \left\{ \underset{w,p}{\mathop {\sup }}\left\langle w,\sum \limits _{k=1}^{n}{\sigma _k\Phi _p\left( {{x}^{\left( k\right) }}\right) }\right\rangle \right\} \\&\le \frac{B}{n}{{\mathbb {E}}_{\sigma }}\left\{ \underset{p}{\mathop {\sup }}\,\sqrt{\left\langle \sum \limits _{k=1}^{n}{{{\sigma }_{k}}{{\Phi }_{p}}\left( {{x}^{\left( k\right) }} \right) },\sum \limits _{l=1}^{n}{{{\sigma }_{l}}{{\Phi }_{p}}\left( {{x}^{\left( l\right) }} \right) } \right\rangle } \right\} \\&=\frac{B}{n}{{\mathbb {E}}_{\sigma }}\left\{ \underset{p}{\mathop {\sup }}\,\sqrt{\sum \limits _{k,l=1}^{n}{{{\sigma }_{k}}{{\sigma }_{l}}\left\langle {{\Phi }_{p}}\left( {{x}^{\left( k\right) }} \right) ,{{\Phi }_{p}}\left( {{x}^{\left( l\right) }} \right) \right\rangle }} \right\} \\&\le \frac{B}{n}\sqrt{{{\mathbb {E}}_{\sigma }}\left\{ \underset{p}{\mathop {\sup }}\,\left[ \sum \limits _{k,l=1}^{n}{{{\sigma }_{k}}{{\sigma }_{l}}\hat{k}_p\left( x^{\left( k\right) },x^{\left( l\right) }\right) }\right] \right\} }, \\ \end{aligned} \end{aligned}$$

(A9)

where $\left\{ \sigma _k\right\} _{k=1}^n$ are i.i.d. Rademacher random variable. The first inequality sign makes use of Cauchy–Schwarz inequality. The last inequality holds due to square root function is concave and can be derived from Jensen’s inequality. As a result,

$$\begin{aligned} {\Psi }_{p}={{\mathbb {E}}_{\sigma }}\left\{ \underset{p}{\mathop {\sup }}\,\left[ \sum \limits _{k,l=1}^{n}{{{\sigma }_{k}}{{\sigma }_{l}}\hat{k}_p\left( x^{\left( k\right) },x^{\left( l\right) }\right) }\right] \right\} . \end{aligned}$$

(A10)

Equation (19) can be proved by combining Eqs. (A8), (A9) and (A10). $\square$