Usage of the GO estimator in high dimensional linear models

Abstract

This paper discusses simultaneous parameter estimation and variable selection and presents a new penalized regression method. The method is based on the idea that the coefficient estimates are shrunken towards a predetermined coefficient vector which represents the prior information. This method can result in smaller length estimates of the coefficients depending on the prior information compared to elastic net. In addition to the establishment of the grouping property, we also show that the new method has the grouping effect when the predictors are highly correlated. Simulation studies and real data example show that the prediction performance of the new method is improved over the well-known ridge, lasso and elastic net regression methods yielding a lower mean squared error and competes about the variable selection under sparse and non-sparse situations.

Appendix

1.1 Proof of Theorem 1

Let

$$\begin{aligned} Q\left( \hat{\varvec{\beta }};{\mathbf {b}},\lambda ,\alpha \right) =\frac{1}{2n}\left\| {\mathbf {y}}-{\mathbf {X}}\varvec{\beta }\right\| _{2}^{2}+\lambda \left( \alpha \left\| \varvec{\beta }\right\| _{1}+\frac{1-\alpha }{2}\left\| \varvec{\beta }-{\mathbf {b}}\right\| _{2}^{2}\right) . \end{aligned}$$

We write the sub-gradients of this function with respect to $\beta _{i}$, $\beta _{j}$ and set them equal to zero:

$$\begin{aligned} \frac{\partial Q}{\partial \beta _{i}}=-\frac{1}{n}{\mathbf {x}}_{i}^{\top }\left( {\mathbf {y}}-{\mathbf {X}}\hat{\varvec{\beta }}\right) +\lambda \alpha {\hat{s}}_{i}+\lambda \left( 1-\alpha \right) \left( {\hat{\beta }}_{i}-b_{i}\right)&=0 \end{aligned}$$

(12)

$$\begin{aligned} \frac{\partial Q}{\partial \beta _{j}}=-\frac{1}{n}{\mathbf {x}}_{j}^{\top }\left( {\mathbf {y}}-{\mathbf {X}}\hat{\varvec{\beta }}\right) +\lambda \alpha {\hat{s}}_{j}+\lambda \left( 1-\alpha \right) \left( {\hat{\beta }}_{j}-b_{j}\right)&=0, \end{aligned}$$

(13)

where ${\hat{s}}_{i}$ and ${\hat{s}}_{j}$ are the sub-gradients of the absolute value function of $\beta _{i}$ and $\beta _{j}$.

Subtracting Eq. (12) from Eq. (13) and applying Cauchy Schwarz inequality, we get

$$\begin{aligned} \left| {\hat{\beta }}_{j}-{\hat{\beta }}_{i}-\left( b_{j}-b_{i}\right) \right| \le \frac{1}{n\lambda \left( 1-\alpha \right) }\sqrt{\left\| {\mathbf {x}}_{i}-{\mathbf {x}}_{j}\right\| _{2}^{2}\left\| \hat{{\mathbf {r}}}\right\| _{2}^{2}}, \end{aligned}$$

(14)

where $\hat{{\mathbf {r}}}={\mathbf {y}}-{\mathbf {X}}\hat{\varvec{\beta }}$. Since $\left\| {\mathbf {x}}_{i}-{\mathbf {x}}_{j}\right\| _{2}^{2}=2\left( 1-\rho \right)$, we obtain

$$\begin{aligned} \left| {\hat{\beta }}_{j}-{\hat{\beta }}_{i}-\left( b_{j}-b_{i}\right) \right| \le \frac{1}{n\lambda \left( 1-\alpha \right) }\sqrt{2\left( 1-\rho \right) \left\| \hat{{\mathbf {r}}}\right\| _{2}^{2}}. \end{aligned}$$

(15)

Furthermore, $Q\left( \hat{\varvec{\beta }};{\mathbf {b}},\lambda ,\alpha \right) \le Q\left( {\mathbf {0}};{\mathbf {b}},\lambda ,\alpha \right)$ holds because $\hat{\varvec{\beta }}$ is the minimizer of Q. Hence, we write

$$\begin{aligned} \frac{1}{2n}\left\| \hat{{\mathbf {r}}}\right\| _{2}^{2}+\lambda \alpha \left\| \hat{\varvec{\beta }}\right\| _{1}+\frac{\lambda \left( 1-\alpha \right) }{2}\left\| \hat{\varvec{\beta }}-{\mathbf {b}}\right\| _{2}^{2}\le \frac{1}{2n}\left\| {\mathbf {y}}\right\| _{2}^{2}+\frac{\lambda \left( 1-\alpha \right) }{2}\left\| {\mathbf {b}}\right\| _{2}^{2} \end{aligned}$$

which implies that

$$\begin{aligned} \left\| \hat{{\mathbf {r}}}\right\| _{2}^{2}\le \left\| {\mathbf {y}}\right\| _{2}^{2}+n\lambda \left( 1-\alpha \right) \left\| {\mathbf {b}}\right\| _{2}^{2}. \end{aligned}$$

(16)

If we consider Eqs. (15) and (16) together, then

$$\begin{aligned} \left| {\hat{\beta }}_{j}-{\hat{\beta }}_{i}-\left( b_{j}-b_{i}\right) \right|&\le \frac{1}{n\lambda \left( 1-\alpha \right) }\sqrt{2\left( 1-\rho \right) }\sqrt{\left\| {\mathbf {y}}\right\| _{1}^{2}+n\lambda \left( 1-\alpha \right) \left\| {\mathbf {b}}\right\| _{2}^{2}} \end{aligned}$$

which completes the proof.$\square$