An Alternating Direction Method with Continuation for Nonconvex Low Rank Minimization

Abstract

In this paper we consider a nonconvex model of recovering low-rank matrices from the noisy measurement. The problem is formulated as a nonconvex regularized least square optimization problem, in which the rank function is replaced by a matrix minimax concave penalty function. An alternating direction method with a continuation (ADMc) technique (on the regularization parameter) is proposed to solve this nonconvex low rank matrix recovery problem. Moreover, under some mild assumptions, the convergence behavior of the alternating direction method for the proposed nonconvex problems is proved. Finally, comprehensive numerical experiments show that the proposed nonconvex model and the ADM algorithm are competitive with the state-of-the-art models and algorithms in terms of efficiency and accuracy.

Appendices

Appendices

1.1 Appendix 1: A Note on Theorem 2.1

In this part, we give some numerical evidences about the assumptions of Theorem 2.1. We choice $m=n=100$, $r=5$, $sr=0.5$, $maxiter= 500$ and $\lambda = 20$. Form the Fig. 4, we find that the $\Vert Y^k\Vert _F$ is always bounded and the $\Vert X^k-Y^k\Vert _F$ is less than $10^{-15}$ after 500 iterations. It implies that the condition $\lim _{k\rightarrow \infty } \Vert Z^{k+1}-Z^{k}\Vert _F= 0$ holds.

Fig. 4

Test results on the $\Vert Y^k\Vert _F$ and $\Vert X^k-Y^k\Vert _F$

1.2 Appendix 2: An Example

We will give an example to show the nonconvex model for the matrix completion problem may not admit a solution if the nonconvex functional is not coercive. Let $m=n=2$, $\Omega = \{(1,1);(1,2);(2,1)\}$, the observation matrix M be given by $M= \left( \begin{array}{ccc} 0 &{} 1 \\ 1 &{} - \\ \end{array} \right)$. We consider the following four nonconvex models:

• (1) $\min \sum _{(i,j)\in \Omega }|X_{i,j} - M_{i,j}|^2, \;\; s.t. \; \text {rank}(X) \le 1$,

• (2) $\min \sum _{(i,j)\in \Omega }|X_{i,j} - M_{i,j}|^2 + \text {rank}(X)$,

• (3) $\min \sum _{(i,j)\in \Omega }|X_{i,j} - M_{i,j}|^2, \;\; s.t. \; \Vert X\Vert _{\lambda ,\tau } \le 1$,

• (4) $\min \sum _{(i,j)\in \Omega }|X_{i,j} - M_{i,j}|^2 + \Vert X\Vert _{\lambda ,\tau }$,

where $\lambda =2$ and $\tau =2$ in the scalar MCP function, then $\rho (t) = \left\{ \begin{array}{l@{\quad }l}2t - \frac{t^2}{4}, &{} |t| < 4 \\ 4, &{} |t|\ge 4\end{array}\right.$ and $\Vert X\Vert _{2,2} = \rho (\sigma _1) + \rho (\sigma _2)$, where $\sigma _1$ and $\sigma _2$ are two singular values of X. Clearly $\rho (t) > t$ for all $0< t < 4$. We will show that problems (1) to (4) have no solutions.

For problem (1), let $X_n = \left( \begin{array}{ll} 1/n &{} 1 \\ 1 &{} n\end{array}\right)$, then we obtain that the object function has the infimum 0. But it is clear that 0 can not be obtained, which implies the nonexistence of solution to problem (1). The similar argument can be applied to show problem (3) does not admit a solution. To see this, firstly $X_n$ defined as above provides a minimum sequence, it remains to show 0 is not reachable. For any $Z = \left( \begin{array}{ll} 0 &{} 1 \\ 1 &{} c \end{array}\right)$, it has two nonzero singular values $\sigma _1 \ge 1\ge \sigma _2 >0$, which implies that $\Vert Z\Vert _{\lambda ,\tau } \ge \min (4,\sigma _1) + \sigma _2 >1$. Therefore problem (3) has no solution.

For problem (2), we first notice that the cost functional has a lower bounded 1 and can not obtain this value. Then $X_n$ as above implies that 1 is the exact lower bound, and hence the nonexistence of solution. To see problem (4), we only need to show 1 is its unreachable exact lower bound. To see this, let the cost function of problem (4) be f(X). For any $Z = \left( \begin{array}{ll} a &{} b \\ c &{} d \end{array}\right)$, we can compute $Z^tZ = \left( \begin{array}{ll} a^2 + b^2 &{} ac+bd \\ ac+bd &{} c^2+d^2 \end{array}\right)$, and hence two eigenvalues $\lambda _1$, $\lambda _2$ are positive and satisfy

$$\begin{aligned} \lambda _1 + \lambda _2 = a^2 + b^2 +c^2 + d^2,\quad \lambda _1\lambda _2 = (ad - bd)^2. \end{aligned}$$

From the definition of scalar MCP function, we have

$$\begin{aligned} \Vert Z\Vert _{\lambda ,\tau } \ge \min (\sigma _1,4) + \min (\sigma _2,4), \end{aligned}$$

where $\sigma _1$ and $\sigma _2$ are two singular values of Z (i.e., $\sigma _i = \sqrt{\lambda _i}$). If the sum of two singular value great than or equal to 1, then $f(Z) \ge 1$ and the equality never happens (since when $1\ge \sigma _1 >0$, the inequality $\rho (\sigma _1) > \sigma _1$ holds). Otherwise let us assume the sum of two singular values is less than 1, we have

$$\begin{aligned} f(Z) \ge \sigma _1 + \sigma _2 + a^2 + (b-1)^2 + (c-1)^2 \ge \lambda _1 + \lambda _2 + a^2 + (b-1)^2 + (c-1)^2. \end{aligned}$$

By observing $b^2 + (1-b)^2 \ge 1/2$, $c^2 + (1-c)^2 \ge 1/2$, we obtain $f(Z) \ge 1$ and the equality can not be obtained (otherwise $a=0$, $b=c=1/2$, $\sigma _1 = \sigma _2 = 0$, which is a contradiction).

One may also find the following two minimization problems

• (5) $\min \text {rank}(X) , \;\; s.t. \;\sum _{(i,j)\in \Omega }|X_{i,j} - M_{i,j}|^2\le \delta$,

• (6) $\min \Vert X\Vert _{\lambda ,\tau }, \;\; s.t. \;\sum _{(i,j)\in \Omega }|X_{i,j} - M_{i,j}|^2\le \delta$

have solutions, but the solutions are unstable with respect to noise level $\delta$.

In general, if the desirable matrix is not a low rank matrix, its low-rank approximation is either not exist or not stable. This explains that the assumptions in Theorem 2.1 are necessary in general, to avoid the possible non-stable computation. It also explains that some existing matrix completion algorithms work well for easy problem ($p/dr \ge 3$) but may be not so efficient for hard problem. On the other hand, if we are interesting in some local minimizers, it may exist and stable. From our numerical experiments, it seems that the proposed ADMc converges to some stable local minimizer and hence it works well for both easy and hard problems.