Reducing Effects of Bad Data Using Variance Based Joint Sparsity Recovery

Abstract

Much research has recently been devoted to jointly sparse (JS) signal recovery from multiple measurement vectors using $\ell _{2,1}$ regularization, which is often more effective than performing separate recoveries using standard sparse recovery techniques. However, JS methods are difficult to parallelize due to their inherent coupling. The variance based joint sparsity (VBJS) algorithm was recently introduced in Adcock et al. (SIAM J Sci Comput, submitted). VBJS is based on the observation that the pixel-wise variance across signals convey information about their shared support, motivating the use of a weighted$\ell _1$ JS algorithm, where the weights depend on the information learned from calculated variance. Specifically, the $\ell _1$ minimization should be more heavily penalized in regions where the corresponding variance is small, since it is likely there is no signal there. This paper expands on the original method, notably by introducing weights that ensure accurate, robust, and cost efficient recovery using both $\ell _1$ and $\ell _2$ regularization. Moreover, this paper shows that the VBJS method can be applied in situations where some of the measurement vectors may misrepresent the unknown signals or images of interest, which is illustrated in several numerical examples.

Proof of Lemma 1

Proof

[Lemma 1] Following the technique described in [22] for the non-weighted, one-dimensional case, let $x\in \mathbb {R}^{N\times N}$ and $w_{i,j} \ge 0$ for all $i,j = 1,\ldots ,N$. We drop the vec notation for simplicity.

Define the objective function $H:\mathbb {R}^{N\times N} \rightarrow \mathbb {R}^{N\times N}$ as

$$\begin{aligned} H(x) := ||x||_{1,w} - \nu ^T(y-x) + \frac{\beta }{2}||y-x||_2^2. \end{aligned}$$

(A.1)

To show H(x) is convex, we first observe that for $\alpha \in (0,1)$ and $p,q\in \mathbb {R}^{N\times N}$, we have

$$\begin{aligned} \begin{aligned}&||y-\alpha p -(1-\alpha )q||_2^2 - \left( \alpha ||y-p||_2^2 +(1-\alpha )||y-q||_2^2\right) \\&\quad =\left( y-\alpha p -(1-\alpha )q\right) ^T\left( y-\alpha p -(1-\alpha )q\right) \\&\qquad -\,\left( \alpha (y-p)^T(y-p) + (1-\alpha )(y-q)^T(y-q)\right) \\&\quad = \alpha (\alpha -1)\left( p^Tp - p^Tq - q^Tp +q^Tq\right) \\&\quad = \alpha (\alpha -1)||p-q||_2^2 \\&\quad \le 0. \end{aligned} \end{aligned}$$

(A.2)

Applying (A.2) to H yields

$$\begin{aligned} \begin{aligned}&H(\alpha p + (1-\alpha )q) -\left( \alpha H(p) + (1-\alpha ) H(q)\right) \\&\quad = ||\alpha p + (1-\alpha )q||_{1,w} - \nu ^T\left( y-(\alpha p +(1-\alpha )q) \right) + \frac{\beta }{2}||y - (\alpha p +(1-\alpha )q)||_2^2 \\&\qquad -\,\alpha ||p||_{1,w} - (1-\alpha )||q||_{1,w} + \alpha \nu ^T(y-p) +(1-\alpha )\nu ^T(y-q) \\&\qquad -\,\frac{\beta \alpha }{2}||y-p||_2^2 - \frac{\beta (1-\alpha )}{2}||y-q||_2^2 \\&\quad \le \frac{\beta }{2}||y - (\alpha p +(1-\alpha )q)||_2^2 -\frac{\beta \alpha }{2}||y-p||_2^2 - \frac{\beta (1-\alpha )}{2}||y-q||_2^2 \\&\quad = \frac{\beta }{2}\alpha (\alpha - 1)||p-q||_2^2 \\&\quad \le 0. \end{aligned} \end{aligned}$$

(A.3)

Therefore H is convex. For $p\ne q$, H is strictly/strongly convex and thus coercive [6, 7, 28]. Hence there exists at least one solution $\hat{x}$ of (4.6), [38].

The subdifferential of $f(x) = ||x||_{1,w}$ is given element-wise as

$$\begin{aligned} \left( \partial _x f(x)\right) _{i,j} = {\left\{ \begin{array}{ll} \text {sign}(x_{i,j})w_{i,j}, &{}\quad x_{i,j} \ne 0 \\ \left\{ h ; |h|\le w_{i,j}, \quad h\in \mathbb {R}\right\} , &{}\quad \text {otherwise}, \end{array}\right. } \end{aligned}$$

(A.4)

where the origin is required to be included according to the optimality condition for convex problems. According to (A.4), to minimize (A.1), each component $\hat{x}_{i,j}$, $i,j = 1,\ldots ,N$, must satisfy

$$\begin{aligned} {\left\{ \begin{array}{ll} \text {sign}({\hat{x}}_{i,j}) w_{i,j} + \beta ({\hat{x}}_{i,j} - y_{i,j}) +\nu _{i,j} = 0, &{}\quad x_{i,j} \ne 0 \\ |v_{i,j} - \beta y_{i,j}| \le w_{i,j}, &{}\quad \text {otherwise} . \end{array}\right. } \end{aligned}$$

(A.5)

If ${\hat{x}}_{i,j}\ne 0$, (A.5) yields

$$\begin{aligned} \frac{w_{i,j}}{\beta }\text {sign}({\hat{x}}_{i,j}) + {\hat{x}}_{i,j} = y_{i,j} - \frac{\nu _{i,j}}{\beta }. \end{aligned}$$

(A.6)

Since $w_{i,j}/\beta >0$, (A.6) implies

$$\begin{aligned} \frac{w_{i,j}}{\beta } + |\hat{x}_{i,j}| = \left| y_{i,j} - \frac{\nu _{i,j}}{\beta }\right| . \end{aligned}$$

(A.7)

Combining (A.6) and (A.7) gives

$$\begin{aligned} \begin{aligned} \text {sign}({\hat{x}}_{i,j})&= \frac{\text {sign}({\hat{x}}_{i,j})|{\hat{x}}_{i,j}| + \text {sign}({\hat{x}}_{i,j})w_{i,j}/\beta }{|{\hat{x}}_{i,j}| + w_{i,j}/\beta } = \frac{\hat{x}_{i,j} + \text {sign}({\hat{x}}_{i,j})w_{i,j}/\beta }{|{\hat{x}}_{i,j}| + w_{i,j}/\beta } \\&= \frac{y_{i,j}-\nu _{i,j}/\beta }{|y_{i,j} - \nu _{i,j}/\beta |} = \text {sign}\left( y_{i,j}-\frac{\nu _{i,j}}{\beta }\right) \end{aligned} \end{aligned}$$

(A.8)

Thus, for ${\hat{x}}_{i,j} \ne 0$, we have

$$\begin{aligned} {\hat{x}}_{i,j} = |{\hat{x}}_{i,j}|\text {sign}({\hat{x}}_{i,j}) = \left( |y_{i,j} - \frac{\nu _{i,j}}{\beta }|-\frac{w_{i,j}}{\beta }\right) \text {sign}\left( y_{i,j} - \frac{\nu _{i,j}}{\beta }\right) , \end{aligned}$$

(A.9)

where we have used (A.7) and (A.8) in the result.

Conversely, we now show that ${\hat{x}}_{i,j} = 0$ if and only if

$$\begin{aligned} \left| y_{i,j} - \frac{\nu _{i,j}}{\beta }\right| \le \frac{w_{i,j}}{\beta }. \end{aligned}$$

(A.10)

First assume that ${\hat{x}}_{i,j} = 0$. Then (A.10) follows from (A.5) since $\beta > 0$.

Now assume (A.10) holds for some ${\hat{x}}_{i,j} \ne 0$. By (A.5), ${\hat{x}}_{i,j}$ satisfies (A.7). Hence

$$\begin{aligned} \left| {\hat{x}}_{i,j}\right| = \left| y_{i,j}-\frac{\nu _{i,j}}{\beta }\right| - \frac{w_{i,j}}{\beta } \le 0 \end{aligned}$$

which only holds for ${\hat{x}}_{i,j} = 0$. Hence by contradiction, ${\hat{x}}_{i,j} = 0$. Combining (A.10) with (A.9) yields

$$\begin{aligned} {\hat{x}}_{i,j} = \max \left\{ |y_{i,j} - \frac{\nu _{i,j}}{\beta }|-\frac{w_{i,j}}{\beta },0\right\} \text {sign}\left( y_{i,j}-\frac{\nu _{i,j}}{\beta }\right) . \end{aligned}$$

which is equivalent to (4.7) in matrix form.