A novel soft subspace clustering algorithm with noise detection for high dimensional datasets

Abstract

When dealing with high dimensional data, clustering faces the curse of dimensionality problem. In such data sets, clusters of objects exist in subspaces rather than in whole feature space. Subspace clustering algorithms have already been introduced to tackle this problem. However, noisy data points present in this type of data can have great impact on the clustering results. Therefore, to overcome these problems simultaneously, the fuzzy soft subspace clustering with noise detection (FSSC-ND) is proposed. The presented algorithm is based on the entropy weighting soft subspace clustering and noise clustering. The FSSC-ND algorithm uses a new objective function and update rules to achieve the mentioned goals and present more interpretable clustering results. Several experiments have been conducted on artificial and UCI benchmark datasets to assess the performance of the proposed algorithm. In addition, a number of cancer gene expression datasets are used to evaluate the performance of the proposed algorithm when dealing with high dimensional data. The results of these experiments demonstrate the superiority of the FSSC-ND algorithm in comparison with the state of the art clustering algorithms developed in earlier research.

Appendix

Proof of Theorem 1

We can rewrite the $J_{\mathrm{FSSC-ND}}$ as follows

$$\begin{aligned} \hbox {J}_{\mathrm{FSSC-ND}} \left( {U,W,V} \right)= & {} \sum _{i=1}^{C+1} {\sum _{j=1}^N {u_{ij} ^{m}} d_{ij}^2 }\nonumber \\&+\,\rho \sum _{i=1}^C {\sum _{k=1}^D {w_{ik} \ln w_{ik} } }\\ \hbox {subject to }\sum _{k=1}^D {w_{ik} }= & {} 1,\quad \forall i\quad \hbox { and} \quad \sum _{i=1}^{C+1} {u_{ij} } =1,\quad \forall j\nonumber \end{aligned}$$

(15)

where, $d_{ij}^2 =\sum \nolimits _{k=1}^D {w_{ik} \left( {x_{jk} -v_{ik} } \right) ^{2}}$ for $i = 1$ to C and $d_{ij}^{2} = \delta ^{2}$ for $i=C+1$; and $u_{ij} =1-\sum \nolimits _{i=1}^C {u_{ij} }$ for $i=C+1$.

Using the Lagrangian multipliers technique the following optimization problem is obtained:

$$\begin{aligned} \Phi _1 \left( {U,\lambda ^{u}} \right) =\sum _{i=1}^{C+1} {\sum _{j=1}^N {u_{ij} ^{m}} d_{ij}^2 } -\sum _{j=1}^N {\lambda _j^u \left( {\sum _{i=1}^C {u_{ij} -1} } \right) } \end{aligned}$$

(16)

where $\lambda ^{u}$ is a vector containing the Lagrange multipliers associated with constraint on U.

Setting the gradient of $\Phi _{1}$ with respect to $u_{ij}$ and $\lambda ^{u}_{j}$ to zero, the optimal value of $u_{ij}$ is obtained.

$$\begin{aligned}&\frac{\partial \Phi _1 }{\partial u_{ij} }=mu_{ij}^{m-1} d_{ij}^2 -\lambda _j^u =0, \end{aligned}$$

(17)

$$\begin{aligned}&u_{ij} =\left( {\frac{\lambda _j^u }{md_{ij}^2 }} \right) ^{1/(m-1)} \end{aligned}$$

(18)

$$\begin{aligned}&\frac{\partial \Phi _1 }{\partial \lambda _j^u }=\sum _{i=1}^{C+1} {u_{ij} } -1=0 \end{aligned}$$

(19)

Substituting $u_{ij}$ derived from Eq. (18) in (19) we have:

$$\begin{aligned} \sum _{i=1}^{C+1} {\left( {\frac{\lambda _j^u }{md_{ij}^2 }} \right) ^{1/(m-1)}}= & {} \left( {\frac{\lambda _j^u }{am}} \right) ^{1/(m-1)}\sum _{i=1}^{C+1} {\left( {\frac{1}{d_{ij}^2 }} \right) ^{1/(m-1)}} \nonumber \\= & {} 1, \end{aligned}$$

(20)

It follows:

$$\begin{aligned} \left( {\frac{\lambda _j^u }{m}} \right) ^{1/(m-1)}=\frac{1}{\sum \nolimits _{i=1}^{C+1} {\left( {\frac{1}{d_{ij}^2 }} \right) ^{1/(m-1)}} } \end{aligned}$$

(21)

Substituting (21) back to (18), we obtain:

$$\begin{aligned} u_{ij} =\frac{1}{\sum \nolimits _{l=1}^{C+1} {\left( {\frac{d_{ij} ^{2}}{d_{lj} ^{2}}} \right) ^{1/(m-1)}} } \end{aligned}$$

(22)

Proof of Theorem 2

Considering the objective function in Eq. (7) and the constraint $\sum \nolimits _{k=1}^D {w_{ik} } =1,\forall i$, the following aggregate function should be minimized according to the Lagrangian multiplier method:

$$\begin{aligned} \Phi _2 \left( {W,\lambda ^{w}} \right)= & {} \sum _{i=1}^C {\sum _{j=1}^N {u_{ij} ^{m}} \sum _{k=1}^D {w_{ik} \left( {x_{jk} -v_{ik} } \right) ^{2}} } \nonumber \\&+\,\rho \sum _{i=1}^C {\sum _{k=1}^D {w_{ik} \ln w_{ik} } }\nonumber \\&-\, \sum _{i=1}^C {\lambda _i^w \left( {\sum _{k=1}^D {w_{ik} } -1} \right) } \end{aligned}$$

(23)

$$\begin{aligned} \frac{\partial \Phi _2 }{\partial w_{ik} }\!= & {} \!\sum _{j=1}^N {u_{ij} ^{m}\left( {x_{jk} -v_{ik} } \right) ^{2}} \!+\!\rho \ln w_{ik} +\rho \!-\!\lambda _i^w \!=\!0\nonumber \\ \end{aligned}$$

(24)

So it follows that:

$$\begin{aligned} w_{ik} =\exp \left( {\frac{\lambda _i^w -\rho }{\rho }} \right) \exp \left( {\frac{-D_{ik} }{\rho }} \right) \end{aligned}$$

(25)

where,

$$\begin{aligned} D_{ik} =\sum _{j=1}^N {u_{ij} ^{m}\left( {x_{jk} -v_{ik} } \right) ^{2}} \end{aligned}$$

(26)

On the other hand:

$$\begin{aligned} \frac{\partial \Phi _2 }{\partial \lambda _i^w }=\sum _{k=1}^D {w_{ik} } -1=0 \end{aligned}$$

(27)

Substituting (25) into (27):

$$\begin{aligned}&\sum _{k=1}^D {w_{ik} } =\exp \left( {\frac{\lambda _i^w -\rho }{\rho }} \right) \sum _{k=1}^D {\exp \left( {\frac{-D_{ik} }{\rho }} \right) } =1 \end{aligned}$$

(28)

$$\begin{aligned}&\exp \left( {\frac{\lambda _i^w -\rho }{\rho }} \right) =\frac{1}{\sum \nolimits _{k=1}^D {\exp \left( {\frac{-D_{ik} }{\rho }} \right) } } \end{aligned}$$

(29)

Substituting (29) back to (25), the final equation for computing $w_{ik}$ is obtained:

$$\begin{aligned} w_{ik} =\frac{\exp \left( {\frac{-D_{ik} }{\rho }} \right) }{\sum \nolimits _{k{^{\prime }}=1}^D {\exp \left( {\frac{-D_{ik^{\prime }} }{\rho }} \right) } } \end{aligned}$$

(30)

Proof of Theorem 3

In order to minimize the objective function, the gradient of $J_\mathrm{FSSC-ND}$ is set to zero:

$$\begin{aligned} \frac{\partial J_\mathrm{FSSC-ND} }{\partial v_{ik} }=-2w_{ik} \sum _{j=1}^N {u_{ij} ^{m}} \left( {x_{jk} -v_{ik} } \right) =0 \end{aligned}$$

(31)

Hence, it will result in the computation formula of $v-_{ik}$ as follows:

$$\begin{aligned} v_{ik} =\frac{\sum \nolimits _{j=1}^N {u_{ij}^m x_{jk} } }{\sum \nolimits _{j=1}^N {u_{ij}^m } } \end{aligned}$$

(32)