Block spectral clustering for multiple graphs with inter-relation

Abstract

Clustering methods for multiple graphs explore and exploit multiple graphs simultaneously to obtain a more accurate and robust partition of the data than that using single graph clustering methods. In this paper, we study the clustering of multiple graphs with inter-relation among vertices in different graphs. The main contribution is to propose and develop a block spectral clustering method for multiple graphs with inter-relation. Our idea is to construct a block Laplacian matrix for multiple graphs and make use of its eigenvectors to perform clustering very efficiently. Global optimal solutions are obtained in the proposed method and they are solutions of relaxation of multiple graphs ratio cut and normalized cut problems. In contrast, existing clustering methods cannot guarantee optimal solutions and their solutions are dependent on initial guesses. Experimental results on both synthetic and real-world data sets are given to demonstrate that the clustering accuracy achieved and computational time required by the proposed block clustering method are better than those by the testing clustering methods in the literature.

Appendix

Proof of Theorem 1

(i) It is clear that $\mathbf{B}$ is symmetric. Given any $\mathbf{f} = [\mathbf{f}_1 \mathbf{f}_2 \cdots \mathbf{f}_M ]^T$ with $\mathbf{f}_m = [ \mathbf{f}_m(1) \mathbf{f}_m(2) \cdots \mathbf{f}_m(N_m) ]^T$, we have

$$\begin{aligned} \mathbf{f}^T \mathbf{B f}= & {} \frac{1}{2} \sum _{m=1}^M \sum _{i,j=1}^{N_m} \mathbf{A}_{m}(i,j) ({\mathbf{f}_m}(i)-{\mathbf{f}_m}(j))^2 \\&+ \beta \sum _{m=1}^{M-1} \sum _{m'=m+1}^{M} \sum _{i=1}^{N_m} \sum _{j=1}^{N_{m'}}{} \mathbf{A}_{m,m'}(i,j) ({\mathbf{f}_m}(i)-{\mathbf{f}_{m'}}(j))^2 \end{aligned}$$

Therefore, $\mathbf{B}$ is semi-positive definite. On the other hand, it is easy to check that $\mathbf{B}{} \mathbf{1} = 0{} \mathbf{1}$ where $\mathbf{1}$ is a vector of all ones.

(ii) We consider $\mathbf{B}$ as a Laplacian matrix for a graph containing $\sum _{m=1}^M N_m$ vertices. It is clear that the number of connected components of this graph is equal to the number of inter-components. Using the spectral graph theory, we know that the multiplicity of the zero eigenvalue of $\mathbf{B}$ is equal to the number of inter-components. $\square$

Proof of Theorem 2

Let us define the following K cluster-indicator $\sum _{m=1}^M N_m$-vectors $\mathbf{y}^{(k)}$ ($k=1,2,\cdots ,K$) as follows

$$\begin{aligned} \mathbf{y}^{(k)}= & {} \left[ \mathbf{y}_1^{(k)}, \mathbf{y}_2^{(k)}, \ldots , \mathbf{y}_m^{(k)} \right] ^T \\ \mathbf{y}_m^{(k)}= & {} \left[ \mathbf{y}_m^{(k)}(1), \mathbf{y}_m^{(k)}(2), \ldots , \mathbf{y}_m^{(k)}(N_m) \right] ^T \end{aligned}$$

with

$$\begin{aligned} \mathbf{y}_m^{(k)}(i) = \left\{ \begin{array}{ll} {\displaystyle \frac{1}{ \sqrt{ M | C_m^{(k)} | } } }, & \quad i \in C_m^{(k)}, \\ 0, & \quad i \notin C_m^{(k)}. \end{array} \right. \end{aligned}$$

(12)

The $\sum _{m=1}^M N_m$-by-K matrix $\mathbf{Y} = [ \mathbf{y}^{(1)}, \mathbf{y}^{(2)}, \ldots , \mathbf{y}^{(K)} ]$ satisfies $\mathbf{Y}^T \mathbf{Y} = \mathbf{I}_K$. Also, $\{ \mathbf{y}_m^{(1)}, \mathbf{y}_m^{(2)}, \ldots , \mathbf{y}_m^{(K)} \}$ are orthogonal for $1 \le m \le M$.

When $C_m^{(k)}\ne \emptyset$, we note that

$$\begin{aligned} (\mathbf{y}_m^{(k)})^T \mathbf{L}_{m} \mathbf{y}_m^{(k)}= & {} \frac{1}{2} \sum _{i,j=1}^{N_m} \mathbf{A}_{m}(i,j) \left( \mathbf{y}_m^{(k)}(i) - \mathbf{y}_m^{(k)}(j) \right) ^2 \\= & {} \frac{\Phi _m ( C_m^{(k)}, \overline{C_m^{(k)}} )}{ M | C_m^{(k)} | } \end{aligned}$$

(13)

for $1 \le k \le K$ and $1 \le m \le M$, and

$$\begin{aligned}&~~~~(\mathbf{y}^{(k)})^T \mathbf{B}_{a} \mathbf{y}^{(k)} \\= & {} \frac{1}{2}\sum _{\begin{array}{c} m,m'=1\\ m \ne m' \end{array}}^{M}\sum _{i=1}^{N_m}\sum _{j=1}^{N_{m'}} \mathbf{A}_{m,m'}(i,j) \left( \mathbf{y}_m^{(k)}(i)-\mathbf{y}_{m'}^{(k)}(j) \right) ^2 \\= & {} \frac{1}{2} \sum _{\begin{array}{c} m,m'=1\\ m \ne m' \end{array}}^{M}\sum _{i=1}^{N_m}\sum _{j=1}^{N_{m'}} \mathbf{A}_{m,m'}(i,j)\left( \mathbf{y}_m^{(k)}(i)^2+ \mathbf{y}_{m'}^{(k)}(j)^2- 2\sum _{i=1}^{N} \mathbf{y}_m^{(k)}(i) \mathbf{y}_{m'}^{(k)}(i)\right) \\= & {} \frac{1}{2}\sum _{\begin{array}{c} m,m'=1\\ m \ne m' \end{array}}^{M}\left( \frac{\Psi _{}( C_m^{(k)},G_{m'})}{ M | C_m^{(k)} |}+\frac{\Psi _{}( C_{m'}^{(k)},G_{m})}{ M | C_{m'}^{(k)}|}-2\frac{\Psi _{}( C_m^{(k)},C_{m'}^{(k)})}{ M \sqrt{ | C_m^{(k)} | | C_{m'}^{(k)} | }}\right) \\= & {} \frac{1}{M}\left( \sum _{m=1}^M\frac{\Upsilon _{}(C_m^{(k)})}{| C_m^{(k)} |}-\sum _{\begin{array}{c} m,m'=1\\ m \ne m' \end{array}}^{M}\frac{\Psi _{}( C_m^{(k)},C_{m'}^{(k)})}{\sqrt{ | C_m^{(k)} | | C_{m'}^{(k)} | }}\right) \end{aligned}$$

(14)

In the first term of (14), $\Upsilon _{a}(C_m^{(k)})$ refers to the across edges linking $C_m^{(k)}$ and all vertices in other relations. By dividing $| C_m^{(k)} |$, it refers to an average inter-degree over each vertex in $C_m^{(k)}$.

When $C_m^{(k)}=\emptyset$, it is easy to verify that the following terms will turn to zero:

$$\begin{aligned} \frac{\Phi _m ( C_m^{(k)}, \overline{C_m^{(k)}} )}{ | C_m^{(k)} | },\quad \frac{\Upsilon _{}(C_m^{(k)})}{| C_m^{(k)} |},\quad \frac{\Psi _{}( C_m^{(k)},C_{m'}^{(k)})}{\sqrt{ | C_m^{(k)} | | C_{m'}^{(k)} | }} \end{aligned}$$

which indicate that our setting in (9) is satisfied.

It follows that $trace( \mathbf{Y}^T \mathbf{B}{} \mathbf{Y} )$ is equal to $J_1( \{C_m^{(1)}, \ldots , C_m^{(K)} \}_{m=1}^{M} )/M$ plus a constant term. This implies optimizing (9) is equivalent to finding a minimum of (6) with a special form of (12). Therefore, (6) is relaxed problem of (9) without imposing a special form. $\square$

Proof of Theorem 3

To prove that

$$\begin{aligned} \{a_1,a_2,\ldots ,a_{N_1}\} \\ \{a_{N_1+1},a_{N_1+2},\ldots ,a_{N_1+N_2}\} \\ \ldots \\ \{a_{N-N_{K-1}+1},a_{N-N_{K-1}+2},\ldots ,a_{N}\} \end{aligned}$$

(15)

is the best partition, we need to show that for any partition

$$\begin{aligned} \{a_{j_1},a_{j_2},\ldots ,a_{{j_{N_1}}}\}\\ \{a_{j_{N_1+1}},a_{j_{N_1+2}},\ldots ,a_{j_{N_1+N_2}}\}\\ \ldots \\ \{a_{j_{N-N_{K-1}+1}},a_{j_{N-N_{K-1}+2}},\ldots ,a_{j_{N}}\} \end{aligned}$$

we have

$$\begin{aligned} \frac{\sum _{k=1}^{N_1}a_k}{N_1}+\frac{\sum _{k=N_1}^{N_1+N_2}a_k}{N_2}+\cdots +\frac{\sum _{k=N-N_{K}+1}^{N}a_k}{N_K}\\&\quad \le \frac{\sum _{k=1}^{N_1}a_{j_k}}{N_1}+\frac{\sum _{k=N_1}^{N_1+N_2}a_{j_k}}{N_2}+\cdots +\frac{\sum _{k=N-N_{K}+1}^{N}a_{j_k}}{N_K} \end{aligned}$$

By substituting

$$\begin{aligned} \sum _{k=N-N_{K}+1}^{N}a_{j_k}=\sum _{k=1}^{N}a_k-\sum _{k=1}^{N_1}a_{j_k}-\cdots -\sum _{k=N-N_{K}-N_{K-1}+1}^{k=N-N_{K}}a_{j_k} \end{aligned}$$

into above inequality, we get

$$\begin{aligned}&\left( \frac{1}{N_1}-\frac{1}{N_K}\right) \sum _{k=1}^{N_1}a_k+\cdots +\left( \frac{1}{N_{K-1}}-\frac{1}{N_K}\right) \sum _{k=N-N_{K}-N_{K-1}+1}^{k=N-N_{K}}a_{k}\\&\quad \le \left( \frac{1}{N_1}-\frac{1}{N_K}\right) \sum _{k=1}^{N_1}a_{j_k}+\cdots +\left( \frac{1}{N_{K-1}}-\frac{1}{N_K}\right) \sum _{k=N-N_{K}-N_{K-1}+1}^{k=N-N_{K}}a_{j_k} \end{aligned}$$

Since $N_1\le N_2\le \cdots \le N_K$, we have $({\frac{1}{N_1}}-{\frac{1}{N_K}})\ge \cdots \ge (\frac{1}{N_{K-1}}-\frac{1}{N_K})\ge 0$, it follows that partition (15) is the best partition. The theorem is proved. $\square$

Proof of Theorem 4

Now we set

$$\begin{aligned} \mathbf{y}_m^{(k)}(i) = \frac{1}{ \sqrt{ M vol(C_m^{(k)}) } } \end{aligned}$$

similar to (12), and we have $(\mathbf{y}_m^{(k)})^T \mathbf{D}_{m,m} \mathbf{y}_m^{(k)} = 1/M$ for $1 \le m \le M$ and $1 \le k \le K$. We can show that

$$\begin{aligned} (\mathbf{y}_m^{(k)})^T \mathbf{L}_{m} \mathbf{y}_m^{(k)} = \frac{\Phi _m ( C_m^{(k)}, \overline{C_m^{(k)}} )}{ M vol(C_m^{(k)}) } \end{aligned}$$

and using the same argument in ratio cut, we obtain

$$\begin{aligned}&( \mathbf{D}^{1/2} \mathbf{y}^{(k)})^T \mathbf{B}_{a} \mathbf{D}^{1/2} \mathbf{y}^{(k)}\\= & {} \frac{1}{2}\sum _{\begin{array}{c} m,m'=1\\ m \ne m' \end{array}}^{M}\sum _{i=1}^{N_m}\sum _{j=1}^{N_{m'}} \mathbf{A}_{m,m'}(i,j) \left( \sqrt{d_i^{(m)}}{} \mathbf{y}_m^{(k)}(i)-\sqrt{d_j^{(m')}}\mathbf{y}_{m'}^{(k)}(j) \right) ^2 \\ \nonumber= & {} \frac{1}{2} \sum _{\begin{array}{c} m,m'=1\\ m \ne m' \end{array}}^{M}\sum _{i=1}^{N_m}\sum _{j=1}^{N_{m'}} \mathbf{A}_{m,m'}(i,j)\left( d_i^{(m)}{} \mathbf{y}_{m}^{(k)}(i)^2+ d_j^{(m')}\mathbf{y}_{m'}^{(k)}(j)^2-2\sum _{i=1}^{N} \mathbf{y}_m^{(k)}(i) \mathbf{y}_{m'}^{(k)}(i) \sqrt{d_i^{(m)} d_j^{(m')}}\right) \\ = & {} \sum _{m=1}^M\frac{\Upsilon '_{}(C_m^{(k)})}{ M vol( C_m^{(k)})}-\sum _{\begin{array}{c} m,m'=1\\ m \ne m' \end{array}}^{M}\frac{\Psi '_{}( C_m^{(k)},C_{m'}^{(k)})}{ M \sqrt{vol( C_m^{(k)})vol( C_{m'}^{(k)})}} \end{aligned}$$

Using the same way to deal with $C_{m}^{(k)}= \emptyset$ or $\ne \emptyset$, it follows that $trace( \mathbf{Y}^T \widehat{\mathbf{B}} \mathbf{Y} )$ is equal to sum of $J_2( \{ C_m^{(1)}, C_m^{(2)}, \ldots , C_m^{(K)} \}_{m=1}^{M} )/M$ and a constant term. Therefore, (8) is a relaxed problem of (11). $\square$