Efficient l _q norm based sparse subspace clustering via smooth IRLS and ADMM

Abstract

Recently, sparse subspace clustering, as a subspace learning technique, has been successfully applied to several computer vision applications, e.g. face clustering and motion segmentation. The main idea of sparse subspace clustering is to learn an effective sparse representation that are used to construct an affinity matrix for spectral clustering. While most of existing sparse subspace clustering algorithms and its extensions seek the forms of convex relaxation, the use of non-convex and non-smooth l _q (0 < q < 1) norm has demonstrated better recovery performance. In this paper we propose an l _q norm based Sparse Subspace Clustering method (lqSSC), which is motivated by the recent work that l _q norm can enhance the sparsity and make better approximation to l ₀ than l ₁. However, the optimization of l _q norm with multiple constraints is much difficult. To solve this non-convex problem, we make use of the Alternating Direction Method of Multipliers (ADMM) for solving the l _q norm optimization, updating the variables in an alternating minimization way. ADMM splits the unconstrained optimization into multiple terms, such that the l _q norm term can be solved via Smooth Iterative Reweighted Least Square (SIRLS), which converges with guarantee. Different from traditional IRLS algorithms, the proposed algorithm is based on gradient descent with adaptive weight, making it well suit for general sparse subspace clustering problem. Experiments on computer vision tasks (synthetic data, face clustering and motion segmentation) demonstrate that the proposed approach achieves considerable improvement of clustering accuracy than the convex based subspace clustering methods.

Appendix: Proof of the Theory 1

To prove Theorem 1, let $u = z^{(j)} - \frac {1}{\rho } w^{(j)}$ , and u is a constant here. Firstly, for j=1,..,d

$$y_{j} = x_{j}^{(k)2} + {\epsilon_{k}^{2}}, \ \ \ z_{j} = x_{j}^{(k+1)2} + \epsilon_{k+1}^{2} $$

Then

$$\begin{array}{@{}rcl@{}} && J(x^{(k)}, \epsilon_{k}) - J(x^{(k+1)}, \epsilon_{k+1}) \\ &&= \sum\limits_{j = 1}^{d} (y_{j}^{q/2} - z_{j}^{q/2}) + \rho/2 (||x^{(k)} - u||_{2}^{2} - ||x^{(k+1)} - u||_{2}^{2}) \end{array} $$

(29)

Note that f(x)=x ^q/2(0 < q < 1) is a concave function, for any y,z∈R ¹

$$ f(y) - f(z) \geq \frac{q}{2} y^{\frac{q}{2} - 1} (y - z) $$

(30)

Using (30), we have

$$\begin{array}{@{}rcl@{}} && J(x^{(k)}, \epsilon_{k}) - J(x^{(k+1)}, \epsilon_{k+1}) \\ && = \sum\limits_{j = 1}^{d} (y_{j}^{q/2} - z_{j}^{q/2}) + \frac{\rho}{2} (||x^{(k)} - u||_{2}^{2} - ||x^{(k+1)} - u||_{2}^{2}) \end{array} $$

(31)

By using the rule of (30), the left part of (31) can be transformed as

$$\begin{array}{@{}rcl@{}} &&\sum\limits_{j = 1}^{d} y_{j}^{q/2} - z_{j}^{q/2} \ge \frac{q}{2} \sum\limits_{j = 1}^{d} W_{jj}^{k} (x_{j}^{(k)2} - x_{j}^{(k+1)2}) \\ &&= \frac{q}{2} \sum\limits_{j = 1}^{d} W_{jj}^{k} (x_{j}^{(k)} - x_{j}^{(k+1)})^{2} + q\sum\limits_{j = 1}^{d} W_{jj}^{(k)} (x_{j}^{(k)} -x_{j}^{(k+1)})x_{j}^{(k+1)} \\ &&=\frac{q}{2} \sum\limits_{j = 1}^{d} W_{jj}^{k} (x_{j}^{(k)} - x_{j}^{(k+1)})^{2} + q(x^{(k)} -x^{(k+1)})'W^{(k)}x^{(k+1)} \end{array} $$

(32)

We now consider the (18), multiplying (x ^(k)−x ^(k+1))^′ on both sides

$$ q(x^{(k)} - x^{(k+1)})^{T}W^{(k)}x^{(k+1)} + \rho(x^{(k)} - x^{(k+1)})'(x^{(k+1)} - u) = 0 $$

(33)

Simplify (32) and get (33), then convert it as follows

$$\begin{array}{@{}rcl@{}} &&\sum\limits_{j = 1}^{d} y_{j}^{q/2} - z_{j}^{q/2} \\ &&\ge \frac{q}{2} \sum\limits_{j = 1}^{d} W_{jj}^{k} (x_{j}^{(k)} - x_{j}^{(k+1)})^{2} - \rho(x^{(k)} - x^{(k+1)})^{T}(x^{(k+1)} - u) \end{array} $$

(34)

Note that f(x)=x ² is a concave function, then for all y,z∈R ^d

$$f(y) - f(z) \geq 2z(y - z) $$

By employing this inequality, we have

$$ ||x^{(k)} - u||_{2}^{2} - ||x^{(k+1)} - u||_{2}^{2} \ge 2(x^{(k)} - x^{(k+1)})'(x^{(k+1)} - u) $$

(35)

Summarized (31), (34), (35)

$$J(x^{(k)}, \epsilon_{k}) - J(x^{(k+1)}, \epsilon_{k+1}) \ge 0 $$

This prove that J(x,𝜖) is an decreasing sequence. Note that

$$ 0<||x^{(k)}||_{q}^{q} \le ||x^{(k)}||_{q,\epsilon_{k}}^{q} \le J(x^{(k)},\epsilon_{k}) \le J(x^{(0)},\epsilon_{0}) $$

(36)

Thus the sequence {x ^(k)} is bounded. Furthermore, if 𝜖>0, the boundedness of {x ^(k))} implies that there exists a subsequence {x ^(k _j )} converging to some point x ^𝜖 _∗, Note that $||x^{(k+1)} - x^{(k)}||_{2} \rightarrow 0$, thus the subsequence x ^(k _j ) also converges to x ^𝜖 _∗. Consider the subsequence in the (18)

$$qW^{(k_{j})}x^{(k_{j})} + \rho (x^{(k_{j})} - u) = 0 $$

Let $k_{j} \rightarrow \infty$, we get

$$ qW^{\epsilon_{*}}x^{\epsilon_{*}} + \rho (x^{\epsilon_{*}} - u)=0 $$

(37)

Therefore, x ^𝜖 _∗ is a critical point of (18).