Skip to main content
Log in

Reducing Effects of Bad Data Using Variance Based Joint Sparsity Recovery

  • Published:
Journal of Scientific Computing Aims and scope Submit manuscript

Abstract

Much research has recently been devoted to jointly sparse (JS) signal recovery from multiple measurement vectors using \(\ell _{2,1}\) regularization, which is often more effective than performing separate recoveries using standard sparse recovery techniques. However, JS methods are difficult to parallelize due to their inherent coupling. The variance based joint sparsity (VBJS) algorithm was recently introduced in Adcock et al. (SIAM J Sci Comput, submitted). VBJS is based on the observation that the pixel-wise variance across signals convey information about their shared support, motivating the use of a weighted\(\ell _1\) JS algorithm, where the weights depend on the information learned from calculated variance. Specifically, the \(\ell _1\) minimization should be more heavily penalized in regions where the corresponding variance is small, since it is likely there is no signal there. This paper expands on the original method, notably by introducing weights that ensure accurate, robust, and cost efficient recovery using both \(\ell _1\) and \(\ell _2\) regularization. Moreover, this paper shows that the VBJS method can be applied in situations where some of the measurement vectors may misrepresent the unknown signals or images of interest, which is illustrated in several numerical examples.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. Although there are subtle differences in the derivations and normalizations, the PA transform can be thought of as higher order total variation (HOTV). Because part of our investigation discusses parameter selection, which depends explicitly on \(||\mathcal {L} f||\), we will exclusively use the PA transform as it appears in [3] so as to avoid any confusion. Explicit formulations for the PA transform matrix can be found in [3]. We also note that the method can be easily adapted for other sparsifying transformations.

  2. We used the Matlab code provided in [14, 42] when implementing (2.6).

  3. For this simple example, each of the \(K = 5\) false measurement vectors was formed by adding a single false data point, with height sampled from the corresponding distribution, (binary, uniform or Gaussian).

  4. Specifically it approximates the jump function \([f](x) = f(x^+) - f(x^-)\) on a set of N grid points.

  5. It was observed in [8] that multiple scales in jump heights can be handled by iteratively redefining a weighted \(\ell _{2,1}\) norm in the MMV case (2.6). This method proved to be computationally expensive, as the optimization problem must be resolved at each iteration, however.

References

  1. Adcock, B., Gelb, A., Song, G., Sui, Y.: Joint sparse recovery based on variances. SIAM J. Sci. Comput. (submitted)

  2. Ao, D., Wang, R., Hu, C., Li, Y.: A sparse SAR imaging method based on multiple measurement vectors model. Remote Sens. 9(3), 297 (2017)

    Article  Google Scholar 

  3. Archibald, R., Gelb, A., Platte, R.B.: Image reconstruction from undersampled Fourier data using the polynomial annihilation transform. J. Sci. Comput. 67(2), 432–452 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  4. Archibald, R., Gelb, A., Yoon, J.: Polynomial fitting for edge detection in irregularly sampled signals and images. SIAM J. Numer. Anal. 43(1), 259–279 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  5. Barzilai, J., Borwein, J.M.: Two-point step size gradient methods. IMA J. Numer. Anal. 8(1), 141–148 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  6. Bauschke, H.H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert Spaces, vol. 408. Springer, New York (2011)

    Book  MATH  Google Scholar 

  7. Beck, A.: Introduction to Nonlinear Optimization: Theory, Algorithms, and Applications with MATLAB. SIAM (2014)

  8. Candes, E.J., Wakin, M.B., Boyd, S.P.: Enhancing sparsity by reweighted \(\ell_1\) minimization. J. Fourier Anal. Appl. 14(5), 877–905 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  9. Chan, T., Marquina, A., Mulet, P.: High-order total variation-based image restoration. SIAM J. Sci. Comput. 22(2), 503–516 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  10. Chartrand, R., Yin, W.: Iteratively reweighted algorithms for compressive sensing. In: IEEE International Conference on Acoustics, Speech and Signal Processing, 2008. ICASSP 2008. pp. 3869–3872. IEEE, (2008)

  11. Chen, J., Huo, X.: Theoretical results on sparse representations of multiple-measurement vectors. IEEE Trans. Signal Process. 54(12), 4634–4643 (2006)

    Article  MATH  Google Scholar 

  12. Cotter, S.F., Rao, B.D., Engan, K., Kreutz-Delgado, K.: Sparse solutions to linear inverse problems with multiple measurement vectors. IEEE Trans. Signal Process. 53(7), 2477–2488 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  13. Daubechies, I., DeVore, R., Fornasier, M., Güntürk, C.S.: Iteratively reweighted least squares minimization for sparse recovery. Commun. Pure Appl. Math. 63(1), 1–38 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  14. Deng, W., Yin, W., Zhang, Y.: Group sparse optimization by alternating direction method. In: Technical Report. Rice University, Houston, TX, Department of Computationa and Applied Mathematics, (2012)

  15. Denker, D., Gelb, A.: Edge detection of piecewise smooth functions from undersampled Fourier data using variance signatures. SIAM J. Sci. Comput. 39(2), A559–A592 (2017)

    Article  MathSciNet  MATH  Google Scholar 

  16. Eldar, Y.C., Mishali, M.: Robust recovery of signals from a structured union of subspaces. IEEE Trans. Inf. Theory 55(11), 5302–5316 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  17. Eldar, Y.C., Rauhut, H.: Average case analysis of multichannel sparse recovery using convex relaxation. IEEE Trans. Inf. Theory 56(1), 505–519 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  18. Fu, W., Li, S., Fang, L., Kang, X., Benediktsson, J.A.: Hyperspectral image classification via shape-adaptive joint sparse representation. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 9(2), 556–567 (2016)

    Article  Google Scholar 

  19. Glowinski, R., Le Tallec, P.: Augmented Lagrangian and operator-splitting methods in nonlinear mechanics. Studies in applied and numerical mathematics, pp. 45–121. SIAM (1989)

  20. Keydel, E.R., Lee, S.W., Moore, J.T.: MSTAR extended operating conditions: a tutorial. In: Aerospace/Defense Sensing and Controls, International Society for Optics and Photonics pp. 228–242 (1996)

  21. Leviatan, D., Temlyakov, V.N.: Simultaneous approximation by greedy algorithms. Adv. Comput. Math. 25(1), 73–90 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  22. Li, C.: An efficient algorithm for total variation regularization with applications to the single pixel camera and compressive sensing. In: Ph.D. Thesis, Citeseer (2009)

  23. Liu, L., Esmalifalak, M., Ding, Q., Emesih, V.A., Han, Z.: Detecting false data injection attacks on power grid by sparse optimization. IEEE Trans. Smart Grid 5(2), 612–621 (2014)

    Article  Google Scholar 

  24. Liu, Q.Y., Zhang, Q., Gu, F.F., Chen, Y.C., Kang, L., Qu, X.Y.: Downward-looking linear array 3D SAR imaging based on multiple measurement vectors model and continuous compressive sensing. J. Sens. 2017, 1–12 (2017)

    Google Scholar 

  25. Liu, Y., Ma, J., Fan, Y., Liang, Z.: Adaptive-weighted total variation minimization for sparse data toward low-dose X-ray computed tomography image reconstruction. Phys. Med. Biol. 57(23), 7923 (2012)

    Article  Google Scholar 

  26. Mishali, M., Eldar, Y.C.: Reduce and boost: Recovering arbitrary sets of jointly sparse vectors. IEEE Trans. Signal Process. 56(10), 4692–4702 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  27. Monajemi, H., Jafarpour, S., Gavish, M., Donoho, D.L., Ambikasaran, S., Bacallado, S., Bharadia, D., Chen, Y., Choi, Y., Chowdhury, M., et al.: Deterministic matrices matching the compressed sensing phase transitions of Gaussian random matrices. Proc. Nat. Acad. Sci. 110(4), 1181–1186 (2013)

    Article  MathSciNet  Google Scholar 

  28. Niculescu, C., Persson, L.E.: Convex functions and their applications: a contemporary approach. Springer, New York (2006)

    Book  MATH  Google Scholar 

  29. Parikh, N., Boyd, S., et al.: Proximal algorithms. Found. Trends Optim. 1(3), 127–239 (2014)

    Article  Google Scholar 

  30. Sanders, T., Gelb, A., Platte, R.B.: Composite SAR imaging using sequential joint sparsity. J. Comput. Phys. 338, 357–370 (2017)

    Article  MathSciNet  Google Scholar 

  31. Singh, A., Dandapat, S.: Weighted mixed-norm minimization based joint compressed sensing recovery of multi-channel electrocardiogram signals. Comput. Electr. Eng. 53, 203–218 (2016)

    Article  Google Scholar 

  32. Steffens, C., Pesavento, M., Pfetsch, M.E.: A compact formulation for the l21 mixed-norm minimization problem. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017, pp. 4730–4734. IEEE, (2017)

  33. Tropp, J.A.: Algorithms for simultaneous sparse approximation. Part II: convex relaxation. Signal Process. 86(3), 589–602 (2006)

    Article  MATH  Google Scholar 

  34. Tropp, J.A., Gilbert, A.C., Strauss, M.J.: Simultaneous sparse approximation via greedy pursuit. In: IEEE International Conference and Proceedings on Acoustics, Speech, and Signal Processing, 2005. (ICASSP’05), vol. 5, pp. v–721. IEEE (2005)

  35. Tropp, J.A., Gilbert, A.C., Strauss, M.J.: Algorithms for simultaneous sparse approximation. Part I: greedy pursuit. Signal Process. 86(3), 572–588 (2006)

    Article  MATH  Google Scholar 

  36. Wang, Y., Yang, J., Yin, W., Zhang, Y.: A new alternating minimization algorithm for total variation image reconstruction. SIAM J. Imaging Sci. 1(3), 248–272 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  37. Wipf, D.P., Rao, B.D.: An empirical bayesian strategy for solving the simultaneous sparse approximation problem. IEEE Trans. Signal Process. 55(7), 3704–3716 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  38. Wright, S., Nocedal, J.: Numerical optimization. Science 35, 67–68 (1999)

    MATH  Google Scholar 

  39. Xie, W., Deng, Y., Wang, K., Yang, X., Luo, Q.: Reweighted l1 regularization for restraining artifacts in fmt reconstruction images with limited measurements. Opt. Lett. 39(14), 4148–4151 (2014)

    Article  Google Scholar 

  40. Yang, Z., Xie, L.: Enhancing sparsity and resolution via reweighted atomic norm minimization. IEEE Trans. Signal Process. 64(4), 995–1006 (2016)

    Article  MathSciNet  Google Scholar 

  41. Ye, F., Luo, H., Lu, S., Zhang, L.: Statistical en-route filtering of injected false data in sensor networks. IEEE J. Sel. Areas Commun. 23(4), 839–850 (2005)

    Article  Google Scholar 

  42. Zhang, Y.: Users guide for YALL1: your algorithms for l1 optimization. In: Technique Report, pp. 09–17 (2009)

  43. Zhao, B., Fei-Fei, L., Xing, E.P.: Online detection of unusual events in videos via dynamic sparse coding. In: 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3313–3320. IEEE (2011)

  44. Zheng, C., Li, G., Liu, Y., Wang, X.: Subspace weighted l21 minimization for sparse signal recovery. EURASIP J. Adv. Signal Process. 2012(1), 98 (2012)

    Article  Google Scholar 

  45. Zhou, F., Wu, R., Xing, M., Bao, Z.: Approach for single channel SAR ground moving target imaging and motion parameter estimation. IET Radar Sonar Navig. 1(1), 59–66 (2007)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Anne Gelb.

Additional information

Anne Gelb’s work is supported in part by the Grants NSF-DMS 1502640, NSF-DMS 1732434, and AFOSR FA9550-15-1-0152. Approved for public release. PA Approval #:[88AWB-2017-6162].

Proof of Lemma 1

Proof of Lemma 1

Proof

[Lemma 1] Following the technique described in [22] for the non-weighted, one-dimensional case, let \(x\in \mathbb {R}^{N\times N}\) and \(w_{i,j} \ge 0\) for all \(i,j = 1,\ldots ,N\). We drop the vec notation for simplicity.

Define the objective function \(H:\mathbb {R}^{N\times N} \rightarrow \mathbb {R}^{N\times N}\) as

$$\begin{aligned} H(x) := ||x||_{1,w} - \nu ^T(y-x) + \frac{\beta }{2}||y-x||_2^2. \end{aligned}$$
(A.1)

To show H(x) is convex, we first observe that for \(\alpha \in (0,1)\) and \(p,q\in \mathbb {R}^{N\times N}\), we have

$$\begin{aligned} \begin{aligned}&||y-\alpha p -(1-\alpha )q||_2^2 - \left( \alpha ||y-p||_2^2 +(1-\alpha )||y-q||_2^2\right) \\&\quad =\left( y-\alpha p -(1-\alpha )q\right) ^T\left( y-\alpha p -(1-\alpha )q\right) \\&\qquad -\,\left( \alpha (y-p)^T(y-p) + (1-\alpha )(y-q)^T(y-q)\right) \\&\quad = \alpha (\alpha -1)\left( p^Tp - p^Tq - q^Tp +q^Tq\right) \\&\quad = \alpha (\alpha -1)||p-q||_2^2 \\&\quad \le 0. \end{aligned} \end{aligned}$$
(A.2)

Applying (A.2) to H yields

$$\begin{aligned} \begin{aligned}&H(\alpha p + (1-\alpha )q) -\left( \alpha H(p) + (1-\alpha ) H(q)\right) \\&\quad = ||\alpha p + (1-\alpha )q||_{1,w} - \nu ^T\left( y-(\alpha p +(1-\alpha )q) \right) + \frac{\beta }{2}||y - (\alpha p +(1-\alpha )q)||_2^2 \\&\qquad -\,\alpha ||p||_{1,w} - (1-\alpha )||q||_{1,w} + \alpha \nu ^T(y-p) +(1-\alpha )\nu ^T(y-q) \\&\qquad -\,\frac{\beta \alpha }{2}||y-p||_2^2 - \frac{\beta (1-\alpha )}{2}||y-q||_2^2 \\&\quad \le \frac{\beta }{2}||y - (\alpha p +(1-\alpha )q)||_2^2 -\frac{\beta \alpha }{2}||y-p||_2^2 - \frac{\beta (1-\alpha )}{2}||y-q||_2^2 \\&\quad = \frac{\beta }{2}\alpha (\alpha - 1)||p-q||_2^2 \\&\quad \le 0. \end{aligned} \end{aligned}$$
(A.3)

Therefore H is convex. For \(p\ne q\), H is strictly/strongly convex and thus coercive [6, 7, 28]. Hence there exists at least one solution \(\hat{x}\) of (4.6), [38].

The subdifferential of \(f(x) = ||x||_{1,w}\) is given element-wise as

$$\begin{aligned} \left( \partial _x f(x)\right) _{i,j} = {\left\{ \begin{array}{ll} \text {sign}(x_{i,j})w_{i,j}, &{}\quad x_{i,j} \ne 0 \\ \left\{ h ; |h|\le w_{i,j}, \quad h\in \mathbb {R}\right\} , &{}\quad \text {otherwise}, \end{array}\right. } \end{aligned}$$
(A.4)

where the origin is required to be included according to the optimality condition for convex problems. According to (A.4), to minimize (A.1), each component \(\hat{x}_{i,j}\), \(i,j = 1,\ldots ,N\), must satisfy

$$\begin{aligned} {\left\{ \begin{array}{ll} \text {sign}({\hat{x}}_{i,j}) w_{i,j} + \beta ({\hat{x}}_{i,j} - y_{i,j}) +\nu _{i,j} = 0, &{}\quad x_{i,j} \ne 0 \\ |v_{i,j} - \beta y_{i,j}| \le w_{i,j}, &{}\quad \text {otherwise} . \end{array}\right. } \end{aligned}$$
(A.5)

If \({\hat{x}}_{i,j}\ne 0\), (A.5) yields

$$\begin{aligned} \frac{w_{i,j}}{\beta }\text {sign}({\hat{x}}_{i,j}) + {\hat{x}}_{i,j} = y_{i,j} - \frac{\nu _{i,j}}{\beta }. \end{aligned}$$
(A.6)

Since \(w_{i,j}/\beta >0\), (A.6) implies

$$\begin{aligned} \frac{w_{i,j}}{\beta } + |\hat{x}_{i,j}| = \left| y_{i,j} - \frac{\nu _{i,j}}{\beta }\right| . \end{aligned}$$
(A.7)

Combining (A.6) and (A.7) gives

$$\begin{aligned} \begin{aligned} \text {sign}({\hat{x}}_{i,j})&= \frac{\text {sign}({\hat{x}}_{i,j})|{\hat{x}}_{i,j}| + \text {sign}({\hat{x}}_{i,j})w_{i,j}/\beta }{|{\hat{x}}_{i,j}| + w_{i,j}/\beta } = \frac{\hat{x}_{i,j} + \text {sign}({\hat{x}}_{i,j})w_{i,j}/\beta }{|{\hat{x}}_{i,j}| + w_{i,j}/\beta } \\&= \frac{y_{i,j}-\nu _{i,j}/\beta }{|y_{i,j} - \nu _{i,j}/\beta |} = \text {sign}\left( y_{i,j}-\frac{\nu _{i,j}}{\beta }\right) \end{aligned} \end{aligned}$$
(A.8)

Thus, for \({\hat{x}}_{i,j} \ne 0\), we have

$$\begin{aligned} {\hat{x}}_{i,j} = |{\hat{x}}_{i,j}|\text {sign}({\hat{x}}_{i,j}) = \left( |y_{i,j} - \frac{\nu _{i,j}}{\beta }|-\frac{w_{i,j}}{\beta }\right) \text {sign}\left( y_{i,j} - \frac{\nu _{i,j}}{\beta }\right) , \end{aligned}$$
(A.9)

where we have used (A.7) and (A.8) in the result.

Conversely, we now show that \({\hat{x}}_{i,j} = 0\) if and only if

$$\begin{aligned} \left| y_{i,j} - \frac{\nu _{i,j}}{\beta }\right| \le \frac{w_{i,j}}{\beta }. \end{aligned}$$
(A.10)

First assume that \({\hat{x}}_{i,j} = 0\). Then (A.10) follows from (A.5) since \(\beta > 0 \).

Now assume (A.10) holds for some \({\hat{x}}_{i,j} \ne 0\). By (A.5), \({\hat{x}}_{i,j}\) satisfies (A.7). Hence

$$\begin{aligned} \left| {\hat{x}}_{i,j}\right| = \left| y_{i,j}-\frac{\nu _{i,j}}{\beta }\right| - \frac{w_{i,j}}{\beta } \le 0 \end{aligned}$$

which only holds for \({\hat{x}}_{i,j} = 0\). Hence by contradiction, \({\hat{x}}_{i,j} = 0\). Combining (A.10) with (A.9) yields

$$\begin{aligned} {\hat{x}}_{i,j} = \max \left\{ |y_{i,j} - \frac{\nu _{i,j}}{\beta }|-\frac{w_{i,j}}{\beta },0\right\} \text {sign}\left( y_{i,j}-\frac{\nu _{i,j}}{\beta }\right) . \end{aligned}$$

which is equivalent to (4.7) in matrix form.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Gelb, A., Scarnati, T. Reducing Effects of Bad Data Using Variance Based Joint Sparsity Recovery. J Sci Comput 78, 94–120 (2019). https://doi.org/10.1007/s10915-018-0754-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10915-018-0754-2

Keywords

Mathematics Subject Classification

Navigation