Minimization of transformed $L_1$ penalty: theory, difference of convex function algorithm, and robust application in compressed sensing

Abstract

We study the minimization problem of a non-convex sparsity promoting penalty function, the transformed $l_1$ (TL1), and its application in compressed sensing (CS). The TL1 penalty interpolates $l_0$ and $l_1$ norms through a nonnegative parameter $a \in (0,+\infty )$, similar to $l_p$ with $p \in (0,1]$, and is known to satisfy unbiasedness, sparsity and Lipschitz continuity properties. We first consider the constrained minimization problem, and discuss the exact recovery of $l_0$ norm minimal solution based on the null space property (NSP). We then prove the stable recovery of $l_0$ norm minimal solution if the sensing matrix A satisfies a restricted isometry property (RIP). We formulated a normalized problem to overcome the lack of scaling property of the TL1 penalty function. For a general sensing matrix A, we show that the support set of a local minimizer corresponds to linearly independent columns of A. Next, we present difference of convex algorithms for TL1 (DCATL1) in computing TL1-regularized constrained and unconstrained problems in CS. The DCATL1 algorithm involves outer and inner loops of iterations, one time matrix inversion, repeated shrinkage operations and matrix-vector multiplications. The inner loop concerns an $l_1$ minimization problem on which we employ the Alternating Direction Method of Multipliers. For the unconstrained problem, we prove convergence of DCATL1 to a stationary point satisfying the first order optimality condition. In numerical experiments, we identify the optimal value $a=1$, and compare DCATL1 with other CS algorithms on two classes of sensing matrices: Gaussian random matrices and over-sampled discrete cosine transform matrices (DCT). Among existing algorithms, the iterated reweighted least squares method based on $l_{1/2}$ norm is the best in sparse recovery for Gaussian matrices, and the DCA algorithm based on $l_1$ minus $l_2$ penalty is the best for over-sampled DCT matrices. We find that for both classes of sensing matrices, the performance of DCATL1 algorithm (initiated with $l_1$ minimization) always ranks near the top (if not the top), and is the most robust choice insensitive to the conditioning of the sensing matrix A. DCATL1 is also competitive in comparison with DCA on other non-convex penalty functions commonly used in statistics with two hyperparameters.

Appendices

Appendix A: Proof of exact TL1 sparse recovery (Theorem 2.1)

Proof

The proof is along the lines of arguments in [4, 9], while using special properties of the penalty function $\rho _a$. For simplicity, we denote $\beta _{C}$ by $\beta$ and $\beta ^0_{C}$ by $\beta ^0$.

Let $e = \beta - \beta ^0$, and we want to prove that the vector $e =0$. It is clear that, $e_{T^c} = \beta _{T^c}$, since T is the support set of $\beta ^0$. By the triangular inequality of $\rho _a$, we have:

$$\begin{aligned} P_a(\beta ^0) - P_a(e_T) = P_a(\beta ^0) - P_a(-e_T) \le P_a(\beta _T). \end{aligned}$$

Then

$$\begin{aligned} P_a(\beta ^0) - P_a(e_T) + P_a(e_{T^c})\le & {} P_a(\beta _T) + P_a(\beta _{T^c}) \\= & {} P_a(\beta ) \\\le & {} P_a(\beta ^0) \\ \end{aligned}$$

It follows that:

$$\begin{aligned} P_a(\beta _{T^c}) = P_a(e_{T^c}) \le P_a(e_T). \end{aligned}$$

(1.1)

Now let us arrange the components at $T^c$ in the order of decreasing magnitude of |e| and partition into L parts: $T^c = T_1 \cup T_2 \cup \cdots \cup T_L$, where each $T_j$ has R elements (except possibly $T_L$ with less). Also denote $T = T_0$ and $T_{01} = T \cup T_1$. Since $Ae = A(\beta - \beta ^0) = 0$, it follows that

$$\begin{aligned} 0= & {} \Vert Ae\Vert _2 \nonumber \\= & {} \Vert A_{T_{01}}e_{T_{01}} + \sum \limits _{j = 2}^L A_{T_j}e_{T_j} \Vert _2 \nonumber \\\ge & {} \Vert A_{T_{01}}e_{T_{01}} \Vert _2 - \sum \limits _{j = 2}^L \Vert A_{T_j}e_{T_j} \Vert _2 \nonumber \\\ge & {} \sqrt{1-\delta _{|T|+R}} \Vert e_{T_{01}}\Vert _2 - \sqrt{1+\delta _{R}} \sum \limits _{j = 2}^L \Vert e_{T_{j}}\Vert _2 \end{aligned}$$

(1.2)

At the next step, we derive two inequalities between the $l_2$ norm and function $P_a$, in order to use the inequality (1.1). Since

$$\begin{aligned} \rho _a(|t|) = \dfrac{(a+1)|t|}{a+|t|}\le & {} \left( \dfrac{a+1}{a}\right) |t| \\= & {} \left( 1 + \dfrac{1}{a}\right) |t| \end{aligned}$$

we have:

$$\begin{aligned} P_a(e_{T_{0}})= & {} \sum \limits _{i \in T_{0}} \rho _a(|e_i|) \nonumber \\\le & {} \left( 1 + \frac{1}{a}\right) \Vert e_{T_{0}} \Vert _1 \nonumber \\\le & {} \left( 1 + \frac{1}{a}\right) \sqrt{|T|} \ \ \Vert e_{T_{0}} \Vert _2 \nonumber \\\le & {} \left( 1 + \frac{1}{a}\right) \sqrt{|T|} \ \ \Vert e_{T_{01}} \Vert _2. \end{aligned}$$

(1.3)

Now we estimate the $l_2$ norm of $e_{T_j}$ from above in terms of $P_a$. It follows from $\beta$ being the minimizer of the problem (2.12) and the definition of $x_C$ (2.9) that

$$\begin{aligned} P_a(\beta _{T^c}) \le P_a(\beta ) \le P_a(x_{C}) \le 1. \end{aligned}$$

For each $i \in T^c,\ \ \rho _a(\beta _i) \le P_a(\beta _{T^c}) \le 1$. Also since

$$\begin{aligned}&\dfrac{(a+1)|\beta _i|}{a+|\beta _i|} \le 1 \nonumber \\&\quad \Leftrightarrow (a+1)|\beta _i| \le a+|\beta _i| \nonumber \\&\quad \Leftrightarrow |\beta _i| \le 1 \end{aligned}$$

(1.4)

we have

$$\begin{aligned} |e_i| = |\beta _i| \le \dfrac{(a+1)|\beta _i|}{a+|\beta _i|} = \rho _a(|\beta _i|) \ \ \ \text{ for } \text{ every } i \in T^c. \end{aligned}$$

It is known that function $\rho _a(t)$ is increasing for non-negative variable $t \ge 0$, and

$$\begin{aligned} |e_i| \le |e_k| \ \textit{ for } \ \forall \ i \in T_j \ \textit{ and } \ \forall \ k \in T_{j-1}, \end{aligned}$$

where $j = 2,3,\ldots ,L$. Thus we have

$$\begin{aligned} |e_i| \le \rho _a(|e_i|)\le & {} P_a(e_{T_{j-1}}) /R \nonumber \\\Rightarrow & {} \Vert e_{T_j}\Vert _2^2 \le \dfrac{P_a(e_{T_{j-1}})^2}{R} \nonumber \\\Rightarrow & {} \Vert e_{T_j}\Vert _2 \le \dfrac{P_a(e_{T_{j-1}})}{R^{1/2}} \nonumber \\\Rightarrow & {} \sum \limits _{j=2}^{L} \Vert e_{T_j}\Vert _2 \le \sum \limits _{j=1}^{L} \dfrac{P_a(e_{T_j})}{R^{1/2}} \end{aligned}$$

(1.5)

Finally, plug (1.3) and (1.5) into inequality (1.2) to get:

$$\begin{aligned} 0\ge & {} \sqrt{1-\delta _{|T|+R}} \dfrac{a}{(a+1)|T|^{1/2}}P_a(e_{T}) - \sqrt{1+\delta _{R}} \dfrac{1}{R^{1/2}} P_a(e_T) \nonumber \\\ge & {} \dfrac{P_a(e_T)}{R^{1/2}} \left( \sqrt{1-\delta _{R+|T|}} \dfrac{a}{a+1} \sqrt{\dfrac{R}{|T|}} - \sqrt{1+\delta _{R}} \right) \end{aligned}$$

(1.6)

By the RIP condition (2.13), the factor $\sqrt{1-\delta _{R+|T|}} \dfrac{a}{a+1} \sqrt{\dfrac{R}{|T|}} - \sqrt{1+\delta _{R}}$ is strictly positive, hence $P_a(e_T) = 0$, and $e_T = 0$. Also by inequality (1.1), $e_{T^c} = 0$. We have proved that $\beta _{C} = \beta ^0_{C}$. The equivalence of (2.12) and (2.11) holds. If another vector $\beta$ is the optimal solution of (2.12), we can prove that it is also equal to $\beta ^0_{C}$, using the same procedure. Hence $\beta _{C}$ is unique. $\square$

Appendix B: Proof of stable TL1 sparse recovery (Theorem 2.2)

Proof

Set $n = A \beta - y_C$. We use three notations below:

• (i) $\beta _{C}^n \Rightarrow$ optimal solution for the constrained problem (2.14);

• (ii) $\beta _{C} \Rightarrow$ optimal solution for the constrained problem (2.12);

• (iii) $\beta _{C}^0 \Rightarrow$ optimal solution for the $l_0$ problem (2.11).

Let T be the support set of $\beta _{C}^0$, i.e., $T = supp(\beta _{C}^0)$, and vector $e = \beta _{C}^n - \beta _{C}^0$. Following the proof of Theorem 2.2, we obtain:

$$\begin{aligned} \sum \limits _{j = 2}^{L} \Vert e_{T_j} \Vert _2 \le \sum \limits _{j = 1}^{L} \frac{P_a(e_{T_j})}{R^{1/2}} = \frac{P_a(e_{T^c})}{R^{1/2}} \end{aligned}$$

and

$$\begin{aligned} \Vert e_{T_{01}} \Vert _2 \ge \frac{a}{(a+1) \sqrt{|T|}} P_a(e_T). \end{aligned}$$

Further, due to the inequality $P_a(\beta _{T^c}^n) = P_a(e_{T^c}) \le P_a(e_T)$ from (1.1) and inequalities in (1.2), we get

$$\begin{aligned} \Vert Ae \Vert _2 \ge \frac{P_a(e_T)}{R^{1/2}} C_{\delta }, \end{aligned}$$

where $C_{\delta } = \sqrt{1-\delta _{R+|T|}} \dfrac{a}{a+1} \sqrt{\dfrac{R}{|T|}} - \sqrt{1+\delta _{R}}$.

By the initial assumption on the size of observation noise, we have

$$\begin{aligned} \Vert Ae \Vert _2 = \Vert A\beta _{C}^n - A \beta _{C}^0 \Vert _2 = \Vert n \Vert _2 \le \tau , \end{aligned}$$

(2.1)

so we have: $P_a(e_T) \le \dfrac{\tau R_{1/2}}{C_{\delta }}$.

On the other hand, we know that $P_a(\beta _{C}) \le 1$ and $\beta _{C}$ is in the feasible set of the problem (2.14). Thus we have the inequality: $P_a(\beta _{C}^n) \le P_a(\beta _{C}) \le 1$. By (1.4), $\beta _{C,i}^n \le 1$ for each i. So, we have

$$\begin{aligned} |\beta _{C,i}^n| \le \rho _a(|\beta _{C,i}^n |). \end{aligned}$$

(2.2)

It follows that

$$\begin{aligned} \Vert e\Vert _2\le & {} \Vert e_T\Vert _2 + \Vert e_{T^c}\Vert _2 = \Vert e_T\Vert _2 + \Vert \beta _{C,T^c}^n\Vert _2 \\\le & {} \dfrac{ \Vert A_T e_T\Vert _2 }{\sqrt{1-\delta _T}} + \Vert \beta _{C,T^c}^n \Vert _1 \\\le & {} \dfrac{ \Vert A_T e_T\Vert _2 }{\sqrt{1-\delta _T}} + P_a(\beta _{C,T^c}^n) = \dfrac{ \Vert A_T e_T\Vert _2 }{\sqrt{1-\delta _T}} + P_a(e_{T^c}) \\\le & {} \dfrac{\tau }{\sqrt{1-\delta _R}} + P_a(e_T) \le D\tau , \end{aligned}$$

for a positive constant D depending only on $\delta _R$ and $\delta _{R+|T|}$. The second inequality uses the definition of RIP, while the first inequality in the last row comes from (2.1) and (1.1). $\square$