Non-negative Sparse Recovery via Momentum-Boosted Adaptive Thresholding Algorithm

Abstract

Recovering a non-negative sparse signal from an underdetermined linear system remains a challenging problem in signal processing. Despite the development of various approaches, such as non-negative least squares, as well as variants of greedy algorithms and iterative thresholding methods, their recovery performance and efficiency often fall short of practical expectations. Aiming to address this limitation, this paper first devises a momentum-boosted adaptive thresholding algorithm for non-negative sparse signal recovery. Then, we establish two sufficient conditions of stable recovery for the proposed algorithm by using the restricted isometry property and mutual coherence. Extensive tests based on synthetic and real-world data demonstrate the superiority of our approach over the state-of-the-art non-negative orthogonal greedy algorithms and iterative thresholding methods, in terms of the probability of successful recovery, phase transition, and computational attractiveness.

Appendices

Proof of Lemma 4

This lemma can be proved using a similar methodology employed in [53, Lemma 3], we provide its proof here for the convenience of readers. Since $\left\Vert {u}\right\Vert _{p} \le \left\Vert {u}\right\Vert _{q}$ holds for any vector ${u}$ and $1\le q \le p\le \infty$, then, to prove (17), we only need to show that

$$\begin{aligned} \left|u_{[K+1]}^{[n]}\right|\le \left\Vert \left( {u}^{[n]}-{\varvec{x}}\right) _{\varOmega }\right\Vert _{\infty }, \end{aligned}$$

where $\varOmega$ is the set of indices of the $K+1$ largest-magnitude entries in ${u}$. Since $|{\text {supp}}\{{\varvec{x}}\}|\le K<|\varOmega |=K+1$, there exists certain index j such that $j\in \varOmega$ and $j\notin {\text {supp}}\{{\varvec{x}}\}$, then we have

$$\begin{aligned} \left|u_{[K+1]}^{[n]}\right|\le \left|u_{j}^{[n]}\right|=\left|u_{j}^{[n]}-x_{j}\right|\le \left\Vert \left( {u}^{[n]}-{\varvec{x}}\right) _{\varOmega }\right\Vert _{\infty }. \end{aligned}$$

Proof of Lemma 5

Proof

According to the Cauchy–Schwarz inequality and $\left\Vert \varvec{A}_{i}\right\Vert _{2}=1$, one can easily show that

$$\begin{aligned} \left\Vert \left( \varvec{A}^{T}\varvec{v}\right) _{\varGamma }\right\Vert _{\infty }=\max _{i\in \{1,\ldots ,M\}}\left|\varvec{A}_{i}^{T}\varvec{v}\right|\le \Vert \varvec{v}\Vert _{2}. \end{aligned}$$

Denote $\varvec{Q}=\varvec{I}-\rho \varvec{A}^{T}\varvec{A}$ and $\mathcal {S} ={\text {supp}}\{{\varvec{x}}\}$. Combining (13) and $\left\Vert \varvec{A}_{i}\right\Vert _{2}=1$ we have $\left|\varvec{A}_{i}^T\varvec{A}_{j}\right|\le \mu ,~i\ne j$, leading to $\left|Q_{ij}\right|\le \max \left\{ |1-\rho |,\mu \rho \right\}$ and

$$\begin{aligned} \left\Vert \left[ \left( \varvec{I}-\rho \varvec{A}^{T}\varvec{A}\right) {\varvec{x}}\right] \right\Vert _{\infty }=\max _{i\in \{1,\ldots ,M\}}\sum \limits _{j\in \mathcal {S}} \left|Q_{ij}x_{j}\right|&\le \max \left\{ |1-\rho |,\mu \rho \right\} \sum \limits _{j\in \mathcal {S}} \left|x_{j}\right|\nonumber \\&=\max \left\{ |1-\rho |,\mu \rho \right\} \Vert {\varvec{x}}\Vert _{1}. \end{aligned}$$

(39)

$\square$

Proof of Lemma 6

Proof

Based on (3), (22), and (23), we can rewrite $\varvec{z}^{[n]}$ in (12) as

$$\begin{aligned} \varvec{z}^{[n]}={\varvec{x}}+\varvec{w}+\varvec{q}. \end{aligned}$$

(40)

Furthermore, by (9) and line 2 of Algorithm 1, we have $\varvec{z}^{[n]}={u}^{[n]}+\bar{{u}}^{[n]}$, where

$$\begin{aligned} u_{i}^{[n]}= {\left\{ \begin{array}{ll} z_{i}^{[n]}, & z_{i}^{[n]}> 0\\ 0, & z_{i}^{[n]} \le 0 \end{array}\right. },~~ \bar{u}_{i}^{[n]}= {\left\{ \begin{array}{ll} 0, & z_{i}^{[n]} > 0\\ z_{i}^{[n]}, & z_{i}^{[n]} \le 0 \end{array}\right. },~~ i\in \{1,2,...,N\}. \end{aligned}$$

(41)

Thus, for any $i\in \{1,\ldots ,N\}$:

• a) If $z_i^{[n]}>0$, it holds that

  $$\begin{aligned} u_i^{[n]}=z_i^{[n]}=x_i+w_i+q_i, \end{aligned}$$

  which implies $u_i^{[n]}-x_i=w_i+q_i$ and hence $\left|u_i^{[n]}-x_i\right|=\left|w_i+q_i\right|$.

• b) If $z_i^{[n]}\le 0$, we have $u_i^{[n]}=0$ and $x_i\le -(w_i+q_i)$. Since $x_i\ge 0$, we have $\left|u_i^{[n]}-x_i\right|=|x_i|\le |w_i+q_i|$.

This implies $\left|u^{[n]}_{i}-x_{i}\right|\le |w_{i}+q_{i}|$ for any $i\in \{1,\ldots ,N\}$, and combining with (17) yield

$$\begin{aligned} \left|u_{[K+1]}^{[n]}\right|\le & \left\Vert ({u}^{[n]}-{\varvec{x}})_{\varOmega }\right\Vert _{\infty } \nonumber \\\le & \left\Vert \varvec{w}+\varvec{q}\right\Vert _{\infty } \nonumber \\\overset{(a)}{\le } & \left\Vert {\varvec{x}}^{[n]}-{\varvec{x}}+ \alpha \varvec{A}^T({\varvec{y}}-\varvec{A}{\varvec{x}}^{[n]}) + \beta ({\varvec{x}}^{[n]}-{\varvec{x}}^{[n-1]})\right\Vert _{\infty } \nonumber \\= & \left\Vert (1+\beta )\left( \varvec{I}-\frac{\alpha }{1+\beta }\varvec{A}^T\varvec{A}\right) ({\varvec{x}}^{[n]}-{\varvec{x}})- \beta \left( {\varvec{x}}^{[n-1]}-{\varvec{x}}\right) + \alpha \varvec{A}^{T}\varvec{v}\right\Vert _{\infty } \nonumber \\\le & (1+\beta ) \left\Vert \left( \varvec{I}-\frac{\alpha }{1+\beta }\varvec{A}^T\varvec{A}\right) \left( {\varvec{x}}^{[n]}-{\varvec{x}}\right) \right\Vert _{\infty } + \beta \left\Vert {\varvec{x}}^{[n-1]}-{\varvec{x}}\right\Vert _{1} + \alpha \left\Vert \varvec{A}^{T}\varvec{v}\right\Vert _{\infty } \nonumber \\\le & \gamma \left\Vert {\varvec{x}}^{[n]}-{\varvec{x}}\right\Vert _{1} + \beta \left\Vert {\varvec{x}}^{[n-1]}-{\varvec{x}}\right\Vert _{1} + \alpha \Vert \varvec{v}\Vert _2, \end{aligned}$$

(42)

where (a) follows from (12) and (40), and the last inequality is due to (18), (19), and (24). Hence (21) holds. $\square$

Proof of Lemma 7

Proof

To simplify notation, we denote

$$\begin{aligned} \varLambda ^{[n]}:= \mathcal {S}^{p^{[n]}}\left( {u}^{[n]}\right) , \end{aligned}$$

(43)

where $\mathcal {S}^{p^{[n]}}(\cdot )$ has been defined in (6).

According to the definitions of $\mathcal {S}^{[n+1]}$ and $\mathcal {S}$, the left-hand side of (25) is rewritten as:

$$\begin{aligned}&\left\Vert {\varvec{x}}^{[n+1]}-{\varvec{x}}\right\Vert _{2}^2 = \left\Vert {\varvec{x}}^{[n+1]}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}\right\Vert _{2}^2 + \left\Vert {\varvec{x}}^{[n+1]}_{\mathcal {S}}-{\varvec{x}}_{\mathcal {S}}\right\Vert _{2}^2, \end{aligned}$$

(44)

$$\begin{aligned}&\left\Vert {\varvec{x}}^{[n+1]}-{\varvec{x}}\right\Vert _{2}^2 = \left\Vert {\varvec{x}}^{[n+1]}_{\mathcal {S}^{[n+1]}}-{\varvec{x}}_{\mathcal {S}^{[n+1]}}\right\Vert _{2}^2 + \left\Vert {\varvec{x}}_{\mathcal {S}\setminus \mathcal {S}^{[n+1]}}\right\Vert _{2}^2. \end{aligned}$$

(45)

Each term of the right-hand side of (44) can be analyzed as follows:

• a) According to (10), (27), (41), and line 4 of Algorithm 1, for any $i \in \mathcal {S}^{[n+1]}$, it holds that

  $$\begin{aligned} {\text {sign}}\left( x^{[n+1]}_{i}\right) = {\text {sign}}\left( u_{i}^{[n]}\right) ,~\left|x^{[n+1]}_{i}\right|\le \left|u_{i}^{[n]}\right|. \end{aligned}$$

  Then, we have

  $$\begin{aligned} \left\Vert {u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}^{[n]}\right\Vert _{2}^2&= \left\Vert {\varvec{x}}^{[n+1]}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}+{u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}^{[n]}-{\varvec{x}}^{[n+1]}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}\right\Vert _{2}^2 \nonumber \\&\ge \left\Vert {\varvec{x}}^{[n+1]}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}\right\Vert _{2}^2 + \left\Vert {u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}^{[n]}-{\varvec{x}}^{[n+1]}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}\right\Vert _{2}^2, \end{aligned}$$

  resulting in

  $$\begin{aligned} \left\Vert {\varvec{x}}^{[n+1]}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}\right\Vert _{2}^2 \le \left\Vert {u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}^{[n]}\right\Vert _{2}^2 - \left\Vert {u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}^{[n]}-{\varvec{x}}^{[n+1]}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}\right\Vert _{2}^2. \end{aligned}$$

  (46)

• b) For $\left\Vert {\varvec{x}}^{[n+1]}_{\mathcal {S}}-{\varvec{x}}_{\mathcal {S}}\right\Vert _{2}^2$, we have

  $$\begin{aligned} \left\Vert {\varvec{x}}^{[n+1]}_{\mathcal {S}}-{\varvec{x}}_{\mathcal {S}}\right\Vert _{2}^2&= \left\Vert {\varvec{x}}^{[n+1]}_{\mathcal {S}}-{u}_{\mathcal {S}}^{[n]}+{u}_{\mathcal {S}}^{[n]}-{\varvec{x}}_{\mathcal {S}}\right\Vert _{2}^2 \nonumber \\&\le 2\left( \left\Vert {\varvec{x}}^{[n+1]}_{\mathcal {S}}-{u}_{\mathcal {S}}^{[n]}\right\Vert _{2}^2 + \left\Vert {u}_{\mathcal {S}}^{[n]}-{\varvec{x}}_{\mathcal {S}}\right\Vert _{2}^2\right) , \end{aligned}$$

  (47)

  where the inequality is due to $(a+b)^2\le 2a^2+2b^2$ for any $a,b\in \mathbb {R}$. According to lines 3 and 4 of Algorithm 1, (10), and (27), it holds that

  $$\begin{aligned} \left\Vert {\varvec{x}}^{[n+1]}_{\mathcal {S}}-{u}_{\mathcal {S}}^{[n]}\right\Vert _{2}^2 \le |\mathcal {S}| \max \limits _{i\in \mathcal {S}}\left|x^{[n+1]}_i-u_i^{[n]}\right|^2 \le K \left|u_{[K+1]}^{[n]}\right|^2. \end{aligned}$$

  (48)

  Substituting (48) into (47) yields

  $$\begin{aligned} \left\Vert {\varvec{x}}^{[n+1]}_{\mathcal {S}}-{\varvec{x}}_{\mathcal {S}}\right\Vert _{2}^2 \le 2K \left|u_{[K+1]}^{[n]}\right|^2+2\left\Vert {u}_{\mathcal {S}}^{[n]}-{\varvec{x}}_{\mathcal {S}}\right\Vert _{2}^2. \end{aligned}$$

  (49)

Then, substituting (46) and (49) into (44) results in

$$\begin{aligned} \left\Vert {\varvec{x}}^{[n+1]}-{\varvec{x}}\right\Vert _{2}^2 \le&\left\Vert {u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}^{[n]}\right\Vert _{2}^2 + 2\left\Vert {u}_{\mathcal {S}}^{[n]}-{\varvec{x}}_{\mathcal {S}}\right\Vert _{2}^2 \nonumber \\&- \left\Vert {u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}^{[n]}-{\varvec{x}}^{[n+1]}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}\right\Vert _{2}^2 + 2K \left|u_{[K+1]}^{[n]}\right|^2 \nonumber \\ =&2\left\Vert {u}_{\mathcal {S}^{[n+1]}\cup \mathcal {S}}^{[n]}-{\varvec{x}}_{\mathcal {S}^{[n+1]}\cup \mathcal {S}}\right\Vert _{2}^2 - \left\Vert {u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}^{[n]}\right\Vert _{2}^2 \nonumber \\&- \left\Vert {u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}^{[n]}-{\varvec{x}}^{[n+1]}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}\right\Vert _{2}^2 + 2K \left|u_{[K+1]}^{[n]}\right|^2 , \end{aligned}$$

(50)

where the equality is due to $\mathcal {S}^{[n+1]}\cup \mathcal {S}= \left( \mathcal {S}^{[n+1]}{\setminus }\mathcal {S}\right) \cup \mathcal {S}$.

According to lines 3 and 4 of Algorithm 1, (10), (27), and (43), it holds that

$$\begin{aligned}&\left\Vert {u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}^{[n]}\right\Vert _{2}^2 \ge \left|\mathcal {S}^{[n+1]}\setminus \mathcal {S}\right|\left|u_{[K+1]}^{[n]}\right|^2, \end{aligned}$$

(51)

$$\begin{aligned}&\left\Vert {u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}^{[n]}-{\varvec{x}}^{[n+1]}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}\right\Vert _{2}^2 = \left|\mathcal {S}^{[n+1]}\setminus \left( \mathcal {S}\cup \varLambda ^{[n]}\right) \right|\left|u_{[K+1]}^{[n]}\right|^2. \end{aligned}$$

(52)

Then, substituting (51) and (52) into (50) results in

$$\begin{aligned} \left\Vert {\varvec{x}}^{[n+1]}-{\varvec{x}}\right\Vert _{2}^2 \le&2\left\Vert {u}_{\mathcal {S}^{[n+1]}\cup \mathcal {S}}^{[n]}-{\varvec{x}}_{\mathcal {S}^{[n+1]}\cup \mathcal {S}}\right\Vert _{2}^2 \nonumber \\&+ \left( 2K-\left|\mathcal {S}^{[n+1]}\setminus \mathcal {S}\right|- \left|\mathcal {S}^{[n+1]}\setminus \left( \mathcal {S}\cup \varLambda ^{[n]}\right) \right|\right) \left|u_{[K+1]}^{[n]}\right|^2 \nonumber \\ \le&\left( 2K+2-\left|\mathcal {S}^{[n+1]}\setminus \mathcal {S}\right|\right. \nonumber \\&\left. -\left|\mathcal {S}^{[n+1]}\setminus \left( \mathcal {S}\cup \varLambda ^{[n]}\right) \right|\right) \left\Vert ({u}^{[n]}-{\varvec{x}})_{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}}\right\Vert _{2}^2, \end{aligned}$$

(53)

where the last inequality is due to (17), (27), and $\mathcal {S}^{[n+1]}={\text {supp}}\left\{ {\varvec{x}}^{[n+1]}\right\}$ is also the set of indices of the K largest-magnitude entries in ${u}^{[n]}$.

For the right-hand side of (45), we carry out the following derivation:

$$\begin{aligned} & \left\Vert {\varvec{x}}^{[n+1]}-{\varvec{x}}\right\Vert _{2}^2 \nonumber \\ & \quad = \left\Vert {\varvec{x}}_{\mathcal {S}^{[n+1]}}^{[n+1]}-{\varvec{x}}_{\mathcal {S}^{[n+1]}}\right\Vert _{2}^2 + \left\Vert {\varvec{x}}_{\mathcal {S}\setminus \mathcal {S}^{[n+1]}}\right\Vert _{2}^2 \nonumber \\ & \quad \le 2\left\Vert {u}_{\mathcal {S}^{[n+1]}}^{[n]}-{\varvec{x}}^{[n+1]}_{\mathcal {S}^{[n+1]}}\right\Vert _{2}^2+2\left\Vert {u}_{\mathcal {S}^{[n+1]}}^{[n]}-{\varvec{x}}_{\mathcal {S}^{[n+1]}}\right\Vert _{2}^2 \nonumber \\ & \qquad + 2\left\Vert {u}_{\mathcal {S}\setminus \mathcal {S}^{[n+1]}}^{[n]}-{\varvec{x}}_{\mathcal {S}\setminus \mathcal {S}^{[n+1]}}\right\Vert _{2}^2 + 2\left\Vert {u}_{\mathcal {S}\setminus \mathcal {S}^{[n+1]}}^{[n]}\right\Vert _{2}^2 \nonumber \\ & \quad = 2 \left( \left\Vert {u}_{\mathcal {S}^{[n+1]}\cup \mathcal {S}}^{[n]}-{\varvec{x}}_{\mathcal {S}^{[n+1]}\cup \mathcal {S}}\right\Vert _{2}^2 + \left\Vert {u}_{\mathcal {S}^{[n+1]}}^{[n]}-{\varvec{x}}^{[n+1]}_{\mathcal {S}^{[n+1]}}\right\Vert _{2}^2 + \left\Vert {u}_{\mathcal {S}\setminus \mathcal {S}^{[n+1]}}^{[n]}\right\Vert _{2}^2\right) , \end{aligned}$$

(54)

where the inequality is due to $(a+b)^2\le 2a^2+2b^2$ for any $a,b\in \mathbb {R}$, and the last equality follows from $\mathcal {S}^{[n+1]}\cup \mathcal {S}=\mathcal {S}^{[n+1]}\cup \left( \mathcal {S}\setminus \mathcal {S}^{[n+1]}\right)$.

According to lines 3 and 4 of Algorithm 1, (27), and (43), it holds that

$$\begin{aligned}&\left\Vert {u}_{\mathcal {S}^{[n+1]}}^{[n]}-{\varvec{x}}^{[n+1]}_{\mathcal {S}^{[n+1]}}\right\Vert _{2}^2 = \left( \left|\mathcal {S}^{[n+1]}\right|-\left|\varLambda ^{[n]}\right|\right) \left|u_{[K+1]}^{[n]}\right|^2 =\left( K-\left|\varLambda ^{[n]}\right|\right) \left|u_{[K+1]}^{[n]}\right|^2, \\&\left\Vert {u}_{\mathcal {S}\setminus \mathcal {S}^{[n+1]}}^{[n]}\right\Vert _{2}^2 \le \left\Vert {u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}^{[n]}\right\Vert _{2}^2 =\left\Vert {u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}^{[n]}-{\varvec{x}}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}\right\Vert _{2}^2. \end{aligned}$$

Substituting them into (54) yields

$$\begin{aligned} \left\Vert {\varvec{x}}^{[n+1]}-{\varvec{x}}\right\Vert _{2}^2 \le&2\left( \left\Vert {u}_{\mathcal {S}^{[n+1]}\cup \mathcal {S}}^{[n]}-{\varvec{x}}_{\mathcal {S}^{[n+1]}\cup \mathcal {S}}\right\Vert _{2}^2 + \left( K-\left|\varLambda ^{[n]}\right|\right) \left|u_{[K+1]}^{[n]}\right|^2 \right. \nonumber \\&+ \left. \left\Vert {u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}^{[n]}-{\varvec{x}}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}\right\Vert _{2}^2\right) \nonumber \\ \overset{(a)}{\le }\ &2\Bigg ( \left\Vert {u}_{\mathcal {S}^{[n+1]}\cup \mathcal {S}}^{[n]}-{\varvec{x}}_{\mathcal {S}^{[n+1]}\cup \mathcal {S}}\right\Vert _{2}^2+\left\Vert {u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}^{[n]}-{\varvec{x}}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}\right\Vert _{2}^2 \nonumber \\&+ \left( K-\left|\varLambda ^{[n]}\right|\right) \left\Vert \left( {u}^{[n]}-{\varvec{x}}\right) _{\mathcal {S}^{[n+1]}\cup {\text {supp}}\left\{ {u}^{[n]}_{[K+1]}\right\} }\right\Vert _{2}^2 \Bigg ) \nonumber \\ \le&\left[ 2K+4-2\left|\varLambda ^{[n]}\right|\right] \left\Vert {u}_{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}}^{[n]}-{\varvec{x}}_{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}}\right\Vert _{2}^2, \end{aligned}$$

(55)

where (a) is due to (17) and $\mathcal {S}^{[n+1]}={\text {supp}}\left\{ {\varvec{x}}^{[n+1]}\right\}$ is also the set of indices of the K largest-magnitude entries in ${u}^{[n]}$, and the last inequality follows from (27).

By combining (53) and (55), we have

$$\begin{aligned} \left\Vert {\varvec{x}}^{[n+1]}-{\varvec{x}}\right\Vert _{2}^2&\le \nu ^{[n]}\left\Vert {u}_{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}}^{[n]}-{\varvec{x}}_{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}}\right\Vert _{2}^2 \nonumber \\&\le \nu ^{*}\left\Vert {u}_{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}}^{[n]}-{\varvec{x}}_{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}}\right\Vert _{2}^2, \end{aligned}$$

(56)

where $\nu ^{[n]}$ and $\nu ^{*}$ have been defined in (26). Then, we prove (25) by taking the square root on both sides of (56). $\square$

Proof of Theorem 1

Proof

According to (27), it holds that

$$\begin{aligned} \left\Vert {u}_{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}}^{[n]}-{\varvec{x}}_{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}}\right\Vert _{2}^2 = \left\Vert {u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}_{+}^{[n]}}^{[n]}-{\varvec{x}}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}_{+}^{[n]}}\right\Vert _{2}^2 + \left\Vert {u}_{ \mathcal {S}_{+}^{[n]}}^{[n]}-{\varvec{x}}_{ \mathcal {S}_{+}^{[n]}}\right\Vert _{2}^2. \end{aligned}$$

(57)

Now we consider two cases (i.e., $z_{i}^{[n]} > 0$ and $z_{i}^{[n]} \le 0$) for the right-hand side of (57) based on (41):

• a) For any $i\in \{1,\ldots ,N\}$, if $z_{i}^{[n]} > 0$, we have $u_{i}^{[n]}=z_{i}^{[n]}$ and $\bar{u}_{i}^{[n]}=0$, yielding

  $$\begin{aligned} \left|u_{i}^{[n]}-x_{i}\right|^2=\left|z_{i}^{[n]}-x_{i}\right|^2 \overset{(40)}{=}|w_{i}+q_{i}|^2 =|w_{i}+q_{i}|^2-\left|\bar{u}_{i}^{[n]}\right|^2. \end{aligned}$$

  (58)

• b) For any $i\in \mathcal {S}^{[n+1]}{\setminus } \mathcal {S}_{+}^{[n]}$, if $z_{i}^{[n]} \le 0$, we have $u_{i}^{[n]}=0$ and $\bar{u}_{i}^{[n]}=z_{i}^{[n]}$, resulting in

  $$\begin{aligned} \left|u_{i}^{[n]}-x_{i}\right|^2\overset{(a)}{=} & 0 \nonumber \\= & |w_{i}+q_{i}|^2-|w_{i}+q_{i}|^2 \nonumber \\\overset{(40)}{=} & |w_{i}+q_{i}|^2-\left|z_{i}^{[n]}-x_{i}\right|^2 \nonumber \\= & |w_{i}+q_{i}|^2-\left|z_{i}^{[n]}\right|^2 \nonumber \\= & |w_{i}+q_{i}|^2-\left|\bar{u}_{i}^{[n]}\right|^2, \end{aligned}$$

  (59)

  where (a) is due to $u_{i}^{[n]}=0$ and $x_{i}=0$.

  Furthermore, by (27), we have $\mathcal {S}_{+}^{[n]}=\left( {\text {supp}}\left\{ {u}^{[n]}_{[K+1]}\right\} {\setminus } \mathcal {S}\right) \cup \mathcal {S}$.

  Then, for $i\in {\text {supp}}\left\{ {u}^{[n]}_{[K+1]}\right\} {\setminus } \mathcal {S}$, if $z_{i}^{[n]} \le 0$, it holds that $u_{i}^{[n]}=0$, $\bar{u}_{i}^{[n]}=z_{i}^{[n]}$, and $x_{i}=0$. By using the same derivation as in (59), we attain

  $$\begin{aligned} \left|u_{i}^{[n]}-x_{i}\right|^2=|w_{i}+q_{i}|^2-\left|\bar{u}_{i}^{[n]}\right|^2. \end{aligned}$$

  (60)

  For $i\in \mathcal {S}$, if $z_{i}^{[n]} \le 0$, we have

  $$\begin{aligned} \left|u_{i}^{[n]}-x_{i}\right|^2=\left|x_{i}\right|^2\overset{(a)}{\le } & |w_{i}+q_{i}|^2-|w_{i}+x_{i}+q_{i}|^2 \nonumber \\\overset{(40)}{=} & |w_{i}+q_{i}|^2-\left|z_{i}^{[n]}\right|^2 \nonumber \\= & |w_{i}+q_{i}|^2-\left|\bar{u}_{i}^{[n]}\right|^2, \end{aligned}$$

  (61)

  where (a) comes from:

  $$\begin{aligned} |w_{i}+q_{i}|^2&=|(w_{i}+q_{i}+x_{i})-x_{i}|^2\\&=|w_{i}+q_{i}+x_{i}|^2+|x_{i}|^2-2(w_{i}+q_{i}+x_{i})x_{i}\\&\ge |w_{i}+q_{i}+x_{i}|^2+|x_{i}|^2, \end{aligned}$$

  and the last inequality follows from $x_{i}>0$ and $z_{i}^{[n]}\overset{(40)}{=}w_{i}+q_{i}+x_{i} \le 0$ for $i \in \mathcal {S}$.

Substituting (58), (59), (60), and (61) into (57) results in

$$\begin{aligned} \left\Vert {u}_{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}}^{[n]}-{\varvec{x}}_{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}}\right\Vert _{2}^2 \le&\left\Vert (\varvec{w}+\varvec{q})_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}_{+}^{[n]}}\right\Vert _{2}^2-\left\Vert \bar{{u}}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}_{+}^{[n]}}^{[n]}\right\Vert _{2}^2 \nonumber \\&+ \left\Vert (\varvec{w}+\varvec{q})_{\mathcal {S}_{+}^{[n]}}\right\Vert _{2}^2-\left\Vert \bar{{u}}_{\mathcal {S}_{+}^{[n]}}^{[n]}\right\Vert _{2}^2. \end{aligned}$$

(62)

Then, by combining (25) and (62), we have

$$\begin{aligned} \left\Vert {\varvec{x}}^{[n+1]}-{\varvec{x}}\right\Vert _{2}^2 \le \nu ^{*} \left\Vert (\varvec{w}+\varvec{q})_{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}}\right\Vert _{2}^2, \end{aligned}$$

(63)

where $\nu ^{*}$ is defined in (26).

Taking the square root on both sides of (63), we attain

$$\begin{aligned} \left\Vert {\varvec{x}}^{[n+1]}-{\varvec{x}}\right\Vert _{2}\le & \sqrt{\nu ^{*}}\left\Vert (\varvec{w}+\varvec{q})_{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}}\right\Vert _{2} \nonumber \\\le & \sqrt{\nu ^{*}}\left( \left\Vert \varvec{w}_{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}}\right\Vert _{2}+\left\Vert \varvec{q}_{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}}\right\Vert _{2}\right) \nonumber \\\overset{(a)}{\le } & \sqrt{\nu ^{*}}\left( (1-\alpha + \beta )\left\Vert \left( {\varvec{x}}^{[n] }-{\varvec{x}}\right) _{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}} \right\Vert _{2} \right. \nonumber \\ & \left. + \alpha \left\Vert \left[ \left( \varvec{I}-\varvec{A}^T\varvec{A}\right) \left( {\varvec{x}}^{[n]}-{\varvec{x}}\right) \right] _{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}} \right\Vert _{2} \right. \nonumber \\ & \left. + \beta \left\Vert \left( {\varvec{x}}^{[n-1]}-{\varvec{x}}\right) _{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}} \right\Vert _{2} + \alpha \left\Vert \left( \varvec{A}^T\varvec{v}\right) _{\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}}\right\Vert _{2} \right) \nonumber \\\overset{(b)}{\le } & \sqrt{\nu ^{*}}\left[ \left( |1-\alpha + \beta | + \alpha \delta _{3K+1}\right) \left\Vert {\varvec{x}}^{[n]}-{\varvec{x}}\right\Vert _{2}+\beta \left\Vert {\varvec{x}}^{[n-1]}-{\varvec{x}}\right\Vert _{2}\right. \nonumber \\ & + \left. \alpha \sqrt{1+\delta _{2K+1}}\Vert \varvec{v}\Vert _{2} \right] \nonumber \\\overset{(31)}{=} & \phi \left\Vert {\varvec{x}}^{[n]}-{\varvec{x}}\right\Vert _{2}+\sqrt{\nu ^{*}}\beta \left\Vert {\varvec{x}}^{[n-1]}-{\varvec{x}}\right\Vert _{2}+ \alpha \sqrt{\nu ^{*}\left( 1+\delta _{2K+1}\right) }\Vert \varvec{v}\Vert _{2},\nonumber \\ \end{aligned}$$

(64)

where (a) follows from (22) and (23), and (b) is due to (14), (15), (27), and

$$\begin{aligned} \left|\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}\cup {\text {supp}}\left\{ {\varvec{x}}^{[n]}-{\varvec{x}}\right\} \right|\le 3K+1,~\left|\mathcal {S}^{[n+1]} \cup \mathcal {S}_{+}^{[n]}\right|\le 2K+1. \end{aligned}$$

When $\delta$ fulfills (28), the range of $\beta$ in (29) is well-defined. Furthermore, it can be verified that:

$$\begin{aligned} \frac{\sqrt{\nu ^{*}}(1+2\beta )-1}{\sqrt{\nu ^{*}}(1-\delta _{3K+1})}<1+\beta <\frac{\sqrt{\nu ^{*}}+1}{\sqrt{\nu ^{*}}(\delta _{3K+1}+1)}. \end{aligned}$$

By the above inequality, we have $\frac{\sqrt{\nu ^{*}}(1+2\beta )-1}{\sqrt{\nu ^{*}}(1-\delta _{3K+1})}< \frac{\sqrt{\nu ^{*}}+1}{\sqrt{\nu ^{*}}(\delta _{3K+1}+1)}$, implying that the range of $\alpha$ in (29) is also well-defined. Therefore, for any fixed $\delta$ and $\beta$, there exists certain $\alpha$ such that either $\frac{\sqrt{\nu ^{*}}(1+2\beta )-1}{\sqrt{\nu ^{*}}(1-\delta _{3K+1})}<\alpha \le 1+\beta$ or $1+\beta<\alpha <\frac{\sqrt{\nu ^{*}}+1}{\sqrt{\nu ^{*}}(\delta _{3K+1}+1)}$. Then, we carry out the following derivation:

• a) For $\frac{\sqrt{\nu ^{*}}(1+2\beta )-1}{\sqrt{\nu ^{*}}(1-\delta _{3K+1})}<\alpha \le 1+\beta$, it holds that

  $$\begin{aligned}&\sqrt{\nu ^{*}}(|1-\alpha + \beta | + \alpha \delta _{3K+1})\nonumber \\&\quad =\sqrt{\nu ^{*}}(1 + \beta - \alpha (1-\delta _{3K+1})) \nonumber \\&\quad =\sqrt{\nu ^{*}}\left( 1 + \beta - \frac{\sqrt{\nu ^{*}}(1+2\beta )-1}{\sqrt{\nu ^{*}}(1-\delta _{3K+1})}(1-\delta _{3K+1})\right) \nonumber \\&\quad =1-\sqrt{\nu ^{*}}\beta . \end{aligned}$$

• b) For $1+\beta<\alpha <\frac{\sqrt{\nu ^{*}}+1}{\sqrt{\nu ^{*}}(\delta _{3K+1}+1)}$, it holds that

  $$\begin{aligned}&\sqrt{\nu ^{*}}(|1-\alpha + \beta | + \alpha \delta _{3K+1})\nonumber \\&\quad =\sqrt{\nu ^{*}}(-1 - \beta + \alpha (1+\delta _{3K+1})) \nonumber \\&\quad =\sqrt{\nu ^{*}}\left( -1 - \beta + \frac{\sqrt{\nu ^{*}}+1}{\sqrt{\nu ^{*}}(\delta _{3K+1}+1)}(1+\delta _{3K+1})\right) \nonumber \\&\quad =1-\sqrt{\nu ^{*}}\beta . \end{aligned}$$

Hence, we have $\sqrt{\nu ^{*}}(|1-\alpha + \beta | + \alpha \delta _{3K+1}+\beta )<1$, combining this with (64) and Lemma 3 give (30) and (32). $\square$

Proof of Theorem 2

Proof

We first define $\varvec{h}\in \mathbb {R}^{N}$ as:

$$\begin{aligned} h_i:= \left\{ \begin{aligned}&1, & i \in \mathcal {S}^{[n+1]}\setminus \varLambda ^{[n]}\\&0, & \text {otherwise} \end{aligned}, \right. \end{aligned}$$

(65)

where $\mathcal {S}^{[n+1]}$ and $\varLambda ^{[n]}$ are defined in (27) and (43), respectively.

According to the definition of $\mathcal {S}^{[n+1]}$ and $\mathcal {S}$, we rewrite $\left\Vert {\varvec{x}}^{[n+1]}-{\varvec{x}}\right\Vert _{1}$ as

$$\begin{aligned} \left\Vert {\varvec{x}}^{[n+1]}-{\varvec{x}}\right\Vert _{1}= \left\Vert {\varvec{x}}^{[n+1]}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}- {\varvec{x}}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}\right\Vert _{1}+\left\Vert {\varvec{x}}^{[n+1]}_{\mathcal {S}}-{\varvec{x}}_{\mathcal {S}}\right\Vert _{1}. \end{aligned}$$

(66)

Then, we analyze each term of the right-hand side of (66).

• (a) According to (27), we have

  $$\begin{aligned} & \left\| \varvec{x}^{[n+1]}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}- \varvec{x}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}\right\| _{1}= \left\| \varvec{x}^{[n+1]}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}\right\| _{1} \nonumber \\ & \quad \overset{(a)}{=}\ \left\| {u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}^{[n]}-\left|u_{[K+1]}^{[n]}\right|{\text {sign}}\left( {u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}^{[n]}\right) \varvec{h}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}} \right\| _{1} \nonumber \\ & \quad \overset{(b)}{=}\left\| {u}_{\varLambda ^{[n]} \setminus \mathcal {S}}^{[n]}\right\| _{1}+\left\| {u}_{\mathcal {S}^{[n+1]}\setminus (\mathcal {S}\cup \varLambda ^{[n]})}^{[n]} \right. \nonumber \\ & \qquad \left. -\left|u_{[K+1]}^{[n]}\right|{\text {sign}}\left( {u}_{\mathcal {S}^{[n+1]}\setminus (\mathcal {S}\cup \varLambda ^{[n]})}^{[n]}\right) \varvec{h}_{\mathcal {S}^{[n+1]}\setminus (\mathcal {S}\cup \varLambda ^{[n]})}\right\| _{1} \nonumber \\ & \quad \overset{(c)}{=}\left\| {u}_{\varLambda ^{[n]} \setminus \mathcal {S}}^{[n]}\right\| _{1}+ \left\| {u}_{\mathcal {S}^{[n+1]}\setminus (\mathcal {S}\cup \varLambda ^{[n]})}^{[n]}\right\| _{1} \nonumber \\ & \qquad - \left|\mathcal {S}^{[n+1]}\setminus \left( \mathcal {S}\cup \varLambda ^{[n]}\right) \right|\left|u_{[K+1]}^{[n]}\right|\nonumber \\ & \quad =\left\| {u}_{\mathcal {S}^{[n+1]}\setminus \mathcal {S}}^{[n]}\right\| _{1} - \left| \mathcal {S}^{[n+1]}\setminus \left( \mathcal {S}\cup \varLambda ^{[n]}\right) \right| \left| u_{[K+1]}^{[n]}\right| , \end{aligned}$$

  (67)

  where (a) follows from lines 3 and 4 of Algorithm 1, (43), and (65), (b) and the last equality can be easily verified since $\varLambda ^{[n]} \subset \mathcal {S}^{[n+1]}$, and (c) is due to: for $i\in \mathcal {S}^{[n+1]}{\setminus }\left( \mathcal {S}\cup \varLambda ^{[n]}\right)$, it holds that

  $$\begin{aligned}&\left|u_i^{[n]}-\left|u_{[K+1]}^{[n]}\right|{\text {sign}}\left( u_i^{[n]}\right) h_i\right|\\&\qquad \overset{(65)}{=} {\left\{ \begin{array}{ll} \big | u_i^{[n]}-\big | u_{[K+1]}^{[n]}\big |\big | =u_i^{[n]}-\left|u_{[K+1]}^{[n]}\right|=\left|u_i^{[n]}\right|-\left|u_{[K+1]}^{[n]}\right|, & u_i^{[n]}>0\\ \big | u_i^{[n]}+\big | u_{[K+1]}^{[n]}\big |\big | =-u_i^{[n]}-\left|u_{[K+1]}^{[n]}\right|=\left|u_i^{[n]}\right|-\left|u_{[K+1]}^{[n]}\right|, & u_i^{[n]}<0 \end{array}\right. }. \end{aligned}$$

• b) Based on (27), it holds that

  $$\begin{aligned} \left\Vert \left( {\varvec{x}}^{[n+1]}-{\varvec{x}}\right) _{\mathcal {S}}\right\Vert _{1}\le & \left\Vert \left( {\varvec{x}}^{[n+1]}-{u}^{[n]}\right) _{\mathcal {S}}\right\Vert _{1}+\left\Vert \left( {u}^{[n]}-{\varvec{x}}\right) _{\mathcal {S}}\right\Vert _{1} \nonumber \\\le & \left|\mathcal {S}\setminus \varLambda ^{[n]}\right|\left|u_{[K+1]}^{[n]}\right|+\left\Vert \left( {u}^{[n]}-{\varvec{x}}\right) _{\mathcal {S}}\right\Vert _{1}, \end{aligned}$$

  (68)

  where the last inequality follows from lines 3 and 4 of Algorithm 1 and (43).

Substituting (67) and (68) into (66) results in

$$\begin{aligned} \left\Vert {\varvec{x}}^{[n+1]}-{\varvec{x}}\right\Vert _{1}\le & \left\Vert \left( {u}^{[n]}-{\varvec{x}}\right) _{\mathcal {S}^{[n+1]}\cup \mathcal {S}}\right\Vert _{1} + \left( \left|\mathcal {S}\setminus \varLambda ^{[n]}\right|\right) \left|u_{[K+1]}^{[n]}\right|\nonumber \\\le & \left\Vert (\varvec{w}+\varvec{q})_{\mathcal {S}^{[n+1]}\cup \mathcal {S}}\right\Vert _{1} + K\left|u_{[K+1]}^{[n]}\right|, \end{aligned}$$

(69)

where the last inequality follows from (20) and $\left|\mathcal {S}\setminus \varLambda ^{[n]}\right|\le K$.

For $\Vert (\varvec{w}+\varvec{q})_{\mathcal {S}^{[n+1]}\cup \mathcal {S}}\Vert _{1}$, we have

$$\begin{aligned} \Vert (\varvec{w}+\varvec{q})_{\mathcal {S}^{[n+1]}\cup \mathcal {S}}\Vert _{1}\le & \Vert \varvec{w}_{\mathcal {S}^{[n+1]}\cup \mathcal {S}}\Vert _{1}+\Vert \varvec{q}_{\mathcal {S}^{[n+1]}\cup \mathcal {S}}\Vert _{1} \nonumber \\\overset{(a)}{\le } & 2K(1+\beta )\left\Vert \left( \varvec{I}-\frac{\alpha }{1+\beta }\varvec{A}^T\varvec{A}\right) \left( {\varvec{x}}^{[n]}-{\varvec{x}}\right) \right\Vert _{\infty } \nonumber \\ & + \beta \left\Vert {\varvec{x}}^{[n-1]}-{\varvec{x}}\right\Vert _{1} + 2\alpha K \left\Vert \varvec{A}^{T}\varvec{v}\right\Vert _{\infty } \nonumber \\\le & 2\gamma K \left\Vert {\varvec{x}}^{[n]}-{\varvec{x}}\right\Vert _1 + \beta \left\Vert {\varvec{x}}^{[n-1]}-{\varvec{x}}\right\Vert _1 + 2\alpha K \Vert \varvec{v}\Vert _{2}, \end{aligned}$$

(70)

where (a) follows from (22), (23), and $\left|\mathcal {S}^{[n+1]}\cup \mathcal {S}\right|\le 2K$, and the last inequality is due to (18), (19), and (24).

Substituting (21) and (70) into (69) yields

$$\begin{aligned} \Vert {\varvec{x}}^{[n+1]}-{\varvec{x}}\Vert _{1}\le & 2\gamma K \left\Vert {\varvec{x}}^{[n]}-{\varvec{x}}\right\Vert _1 + \beta \left\Vert {\varvec{x}}^{[n-1]}-{\varvec{x}}\right\Vert _1 + 2\alpha K \Vert \varvec{v}\Vert _{2} \nonumber \\ & +K(\gamma \left\Vert {\varvec{x}}^{[n]}-{\varvec{x}}\right\Vert _{1} + \beta \left\Vert {\varvec{x}}^{[n-1]}-{\varvec{x}}\right\Vert _{1} + \alpha \Vert \varvec{v}\Vert _2) \nonumber \\= & 3\gamma K\Vert {\varvec{x}}^{[n]}-{\varvec{x}}\Vert _1 + (K+1)\beta \left\Vert {\varvec{x}}^{[n-1]}-{\varvec{x}}\right\Vert _1 + 3\alpha K\Vert \varvec{v}\Vert _2, \end{aligned}$$

(71)

where $\gamma$ has been defined in (24).

According to (24), when $\mu$ satisfies (33) while $\alpha$ and $\beta$ fulfill (34), it holds that $\gamma =\mu \alpha$, resulting in

$$\begin{aligned} 3\gamma K=3\alpha \mu K<1-(K+1)\beta , \end{aligned}$$

where the inequality follows from the range of $\alpha$ in (34). This implies $3\alpha \mu K+(K+1)\beta <1$, and combining with (71) and Lemma 3 yield (35) and (36). Hence, we prove Theorem 2. $\square$