Sparsity aware consistent and high precision variable selection

Abstract

Variable selection is fundamental while dealing with sparse signals that contain only a few number of nonzero elements. This is the case in many signal processing areas extending from high-dimensional statistical modeling to sparse signal estimation. This paper explores a new and efficient approach to model a system with underlying sparse parameters. The idea is to get the noisy observations and estimate the minimum number of underlying parameters with acceptable estimation accuracy. The main challenge is due to the non-convex optimization problem to be solved. The reconstruction stage deals with some suitable objective function in order to estimate the original sparse signal by performing variable selection procedure. This paper introduces a suitable objective function in order to simultaneously recover the true support of the underlying sparse signal while still achieving an acceptable estimation error. It is shown that the proposed method performs the best variable selection compared to the other algorithms, while approaching the lowest least mean squared error in almost all the cases.

Appendices

Appendix A

In (7), writing the first two terms of the Taylor series expansion of $\mathcal P _{\tau ,\gamma } \left( {\left| {\theta _{j} } \right|} \right)$ about $\theta _j^o$, as,

$$\begin{aligned} \mathcal P _{\tau ,\gamma } \left( {\left| {\theta _{j} } \right|} \right)\approx \mathcal P _{\tau ,\gamma } \left( {\left| {\theta _{j}^o } \right|} \right)+\mathcal{P }^{\prime }_{\tau ,\gamma } \left( {\left| {\theta _{j}^o } \right|} \right)\left( {\left| {\theta _j } \right|-\left| {\theta _{j}^o } \right|} \right).\nonumber \\ \end{aligned}$$

(36)

Substituting the approximation (36) into (7), the approximated objective function can be considered as,

$$\begin{aligned} \mathcal{J }\left( {\varvec{\uptheta }} \right)&\!\approx \! \frac{1}{2n}\left\Vert {\mathbf{y}\!-\!\mathbf{X}{\varvec{\uptheta }}} \right\Vert_2^2\nonumber \\&\quad \!+\!\sum _{j\!=\!1}^d {\left[ \mathcal{P _{\tau ,\gamma } \left( {\left| {\theta _{j}^o } \right|} \right) +\mathcal{P }^{\prime }_{\tau ,\gamma } \left( {\left| {\theta _j^o } \right|} \right)\left( {\left| {\theta _j } \right|\,-\,\left| {\theta _{j}^o } \right|} \right)} \right]}.\nonumber \\ \end{aligned}$$

(37)

Finally, excluding the constant terms from (37), the estimator ${\hat{{\varvec{\uptheta }}}}$ is obtained as,

$$\begin{aligned} {\hat{{\varvec{\uptheta }}}}=\mathop {\arg \min }\limits _{\varvec{\uptheta }} \;\;\frac{1}{2n}\left\Vert {\mathbf{y}-\mathbf{X}{\varvec{\uptheta }}} \right\Vert_2^2 +\sum _{j=1}^d {\mathcal{P }^{\prime }_{\tau ,\gamma } \left( {\left| {\theta _{j}^o } \right|} \right)\left| {\theta _j } \right|}. \end{aligned}$$

Appendix B

The term $Z^{(n)}(\mathbf{u})-Z^{(n)}(\mathbf{0})$ could be written as follows using (16),

$$\begin{aligned} Z^{(n)}(\mathbf{u})\!-\!Z^{(n)}(\mathbf{0})\, =\, \frac{1}{2}\left\Vert {\mathbf{y}\!-\!\mathbf{X}\!\left( {{\varvec{\uptheta }}^{*}\!+\!\frac{\mathbf{u}}{\sqrt{n}}} \right)}\!\right\Vert_2^2 +\frac{1}{2}\left\Vert {\mathbf{y}\!-\!\mathbf{X}{\varvec{\uptheta }}^{*}} \right\Vert_2^2 \\ +\,n\sum _{j=1}^d {\mathcal{P }^{\prime }_{\tau ,\gamma } \left( {\left| {\theta _j^o } \right|} \right)\!\!\left({\left| {\theta _j^*\!+\!\frac{u_j }{\sqrt{n}}} \right|\!-\!\left| {\theta _j^*} \right|} \right)}. \end{aligned}$$

For the sake of notation simplicity, the rightmost term of the above equation is suppressed in the following, since it will remain unchanged. So, we have the following:

$$\begin{aligned}&\frac{1}{2}\left\Vert {\mathbf{y}-\mathbf{X}\left( {{\varvec{\uptheta }}^{*}+\frac{\mathbf{u}}{\sqrt{n}}} \right)} \right\Vert_2^2 +\frac{1}{2}\left\Vert {\mathbf{y}-\mathbf{X}\varvec{\uptheta } ^{*}} \right\Vert_2^2\\&\quad =\frac{1}{2}\left[ {\mathbf{y}-\mathbf{X}\varvec{\uptheta } -\frac{1}{\sqrt{n}}\mathbf{Xu}} \right]^{T}\left[ {\mathbf{y}-\mathbf{X}\varvec{\uptheta }-\frac{1}{\sqrt{n}}\mathbf{Xu}} \right]\\&\qquad -\frac{1}{2}\left[ {\mathbf{y}-\mathbf{X}\varvec{\uptheta } } \right]^{T}\left[ {\mathbf{y}-\mathbf{X}\varvec{\uptheta }} \right]. \end{aligned}$$

After some manipulations, we will have the following:

$$\begin{aligned}&\frac{1}{2}\left\Vert {\mathbf{y}-\mathbf{X}\left( {{\varvec{\uptheta }}^{*}+\frac{\mathbf{u}}{\sqrt{n}}} \right)} \right\Vert_2^2 +\frac{1}{2}\left\Vert {\mathbf{y}-\mathbf{X}\varvec{\uptheta } ^{*}} \right\Vert_2^2\nonumber \\&\quad =\frac{1}{\sqrt{n}}\mathbf{u}^{T}\mathbf{X}^{T}\mathbf{X}\varvec{\uptheta } ^{*}-\frac{1}{\sqrt{n}}\mathbf{u}^{T}\mathbf{X}^{T}\mathbf{y}+\frac{1}{2n}\mathbf{u}^{T}\mathbf{X}^{T}\mathbf{Xu} \nonumber \\&\quad =\frac{-1}{\sqrt{n}}\mathbf{u}^{T}\mathbf{X}^{T}\left( {\mathbf{y}-\mathbf{X}\varvec{\uptheta } ^{*}} \right)+\frac{1}{2}\mathbf{u}^{T}\left( {\frac{1}{n}\mathbf{X}^{T}\mathbf{X}} \right)\mathbf{u}. \end{aligned}$$

(38)

Substituting $\mathbf{v}=\mathbf{y}-\mathbf{X}\varvec{\uptheta }$ in ($\text{ B}_{1})$, we have

$$\begin{aligned}&\frac{1}{2}\left\Vert {\mathbf{y}-\mathbf{X}\left( {{\varvec{\uptheta }}^{*}+\frac{\mathbf{u}}{\sqrt{n}}} \right)} \right\Vert_2^2 +\frac{1}{2}\left\Vert {\mathbf{y}-\mathbf{X}\varvec{\uptheta }^{*}} \right\Vert_2^2 =\frac{-1}{\sqrt{n}}\mathbf{u}^{T}\mathbf{X}^{T}\mathbf{v}\nonumber \\&\quad +\frac{1}{2}\mathbf{u}^{T}\left( {\frac{1}{n}\mathbf{X}^{T}\mathbf{X}} \right)\mathbf{u}. \end{aligned}$$

(39)

Knowing that $\lim _{n\rightarrow \infty } \frac{1}{n}\mathbf{X}^{T}\mathbf{X}=\mathbf{C}$, the second term in the right hand of ($\text{ B}_{2})$ tends to $\frac{1}{2}\mathbf{u}^{T}\mathbf{Cu}$ as $n$ goes to infinity. By using the Slutsky’s theorem and central limit theorem, it is easy to show that the first term in the right hand of ($\text{ B}_{2})$ tends to a zero-mean normal distribution with covariance matrix of $\sigma ^{2}\mathbf{C}$.