Second-order least-squares estimation for regression models with autocorrelated errors

Abstract

In their recent paper, Wang and Leblanc (Ann Inst Stat Math 60:883–900, 2008) have shown that the second-order least squares estimator (SLSE) is more efficient than the ordinary least squares estimator (OLSE) when the errors are independent and identically distributed with non zero third moments. In this paper, we generalize the theory of SLSE to regression models with autocorrelated errors. Under certain regularity conditions, we establish the consistency and asymptotic normality of the proposed estimator and provide a simulation study to compare its performance with the corresponding OLSE and generalized least square estimator (GLSE). It is shown that the SLSE performs well giving relatively small standard error and bias (or the mean square error) in estimating parameters of such regression models with autocorrelated errors. Based on our study, we conjecture that for less correlated data, the standard errors of SLSE lie between those of the OLSE and GLSE which can be interpreted as adding the second moment information can improve the performance of an estimator.

Appendices

Appendices

1.1 Appendix 1: Proof of Theorem 1

We will show that assumption of the theorem will fulfill the condition of Lemma 3 of Amemiya (1973). Assumption 1 implies that ${Q_N}(\psi )$ is measurable and continuous in $\psi \in \varGamma$ with probability one. Using Assumption 2 and 3 and Cauchy Schwarz inequality we have

$$\begin{aligned} \left| {\left\| {{W_i}} \right\| \mathop {\sup }\limits _\varGamma \left( {{Y_i} - f({\mathbf{{X}}_i},\beta )} \right) ^2} \right| \le 2{\left\| {{W_i}} \right\| ^{1 + \delta }}Y_i^{2 + \delta } + 2{\left\| {{W_i}} \right\| ^{1 + \delta }}\mathop {\sup }\limits _\varOmega {\left| {f({\mathbf{{X}}_i},\beta )} \right| ^{2 + \delta }} \end{aligned}$$

and

$$\begin{aligned}&\left| {\left\| {{W_i}} \right\| \mathop {\sup }\limits _\varGamma {{\left( {Y_i^2 \!-\! {f^2}({\mathbf{{X}}_i},\beta ) \!-\! \gamma (0)} \right) }^2}} \right| \\&\quad \le 3{\left\| {{W_i}} \right\| ^{1 + \delta }}Y_i^{4 + \delta } + 3{\left\| {{W_i}} \right\| ^{1 + \delta }}\mathop {\sup }\limits _\varOmega {\left| {f({\mathbf{{X}}_i},\beta )} \right| ^{4 + \delta }} + 3{\left\| {{W_i}} \right\| ^{1 + \delta }}\mathop {\sup }\limits _\varSigma {({\gamma (0)})^{2 + \delta }} \end{aligned}$$

which imply $\mathop {\sup }\nolimits _\varGamma \left\| \rho _i^{'}(\psi ){W_i}{\rho _i}(\psi ) \right\| \!\!\le \!\! \left\| {{W_i}} \right\| \mathop {\sup }\nolimits _\varGamma {\left\| {{\rho _i}{{(\psi )}}} \right\| }$ where $E\left\| {{W_i}} \right\| ^ {1+\delta }\mathop {\sup }\nolimits _\varGamma {\left\| {{\rho _i}{{(\psi )}}} \right\| ^{2+\delta }} < \infty$ for some $\delta >0$.

It follows from the uniform strong law of large numbers of White and Domowitz (1984), Theorem 2.3. that ${\textstyle {1 \over N}}{Q_N}(\psi )$ and $Q(\psi ) = \frac{1}{N}\sum \nolimits _{i = 1}^N {E(\rho _i^{'}(\psi ){W_i}{\rho _i}(\psi ))}$ converge almost surely and uniformly for all $\psi$ in $\varGamma$ to the same limit, let say ${\bar{Q}}(\psi )$. Furthermore, ${({\rho _i}(\psi ) - {\rho _i}({\psi _0}))}$ does not depend on $Y_i$, we have

$$\begin{aligned} E\left[ {\rho _i^{'}({\psi _0}){W_i}{\rho _i}(\psi )} \right] = E\left[ {E(\rho _i^{'}({\psi _0})|{X_i}){W_i}{\rho _i}(\psi )} \right] = 0 \end{aligned}$$

which implies $Q(\psi ) = Q({\psi _0}) + \frac{1}{N}\sum \nolimits _{i = 1}^N {E\left[ {({\rho _i}(\psi ) \!-\! {\rho _i}({\psi _0})){'}{W_i}({\rho _i}(\psi ) \!-\! {\rho _i}({\psi _0}))} \right] }$. It follows that $Q(\psi ) \ge Q({\psi _0})$ and by Assumption 4, equality holds if and only if $\psi = \psi _0$. Thus by applying Lemma 3 of Amemiya (1973), we have ${{\hat{\psi }}_{SLSE}}\xrightarrow {{a.s}}{\psi _0}$, as $N \rightarrow \infty$.

1.2 Appendix 2: Proof of Theorem 2

The first derivative $\partial Q_{N} (\psi )/\partial (\psi )$ exists by Assumption 5 and it has the first-order Taylor expansion in $\varGamma$. Since ${{\hat{\psi }}_{SLSE}}\xrightarrow {{a.s}}{\psi _0}$ for sufficiently large $N$ we have

$$\begin{aligned} \frac{{\partial {Q_N}({\psi _0})}}{{\partial \psi }} + \frac{{{\partial ^2}{Q_N}({{\tilde{\psi }}_N})}}{{\partial \psi \partial \psi '}}({{\hat{\psi }}_{SLSE}} - {\psi _0}) =\frac{{\partial {Q_N}({{\hat{\psi }}_{SLSE}})}}{{\partial \psi }} = \mathbf{0} \end{aligned}$$

(5)

where $\left\| {{{\tilde{\psi }}_N} - {\psi _0}} \right\| \leqslant \left\| {{{\hat{\psi }}_{SLSE}} - {\psi _0}} \right\|$. The first derivative of $Q_{N}(\psi )$ in Eq. (5) is given by

$$\begin{aligned} \frac{{\partial {Q_N}(\psi )}}{{\partial \psi }} = 2\sum \limits _{i = 1}^N {\frac{{\partial \rho _i^{'}(\psi )}}{{\partial \psi }}} {W_i}{\rho _i}(\psi ) \end{aligned}$$

where

$$\begin{aligned} \frac{{\partial \rho _i^{'}({\psi })}}{{\partial \psi }} = - \left( {\begin{array}{*{20}c} {\frac{{\partial f(\mathbf{X}_i ,\beta )}}{{\partial \beta }}} &{}\quad {2f(\mathbf{X}_i ,\beta )\frac{{\partial f(\mathbf{X}_i ,\beta )}}{{\partial \beta }}} \\ \mathbf{0} &{}\quad {\frac{{\partial \gamma (0)}}{{\partial \mathbf{a}}}} \\ 0 &{}\quad {\frac{{\partial \gamma (0)}}{ {\partial \sigma ^2 }}} \\ \end{array}} \right) \end{aligned}$$

(6)

The second derivative of $Q_N(\psi )$ in Eq. (5) is given by

$$\begin{aligned} \frac{{{\partial ^2}{Q_N}(\psi )}}{{\partial \psi \partial \psi {'}}} = 2\sum \limits _{i = 1}^N {\left[ {\frac{{\partial \rho _i^{'}(\psi )}}{{\partial \psi }}{W_i}\frac{{\partial {\rho _i}(\psi )}}{{\partial \psi {'}}} + (\rho _i^{'}(\psi ){W_i} \otimes I_{q+p+1}) \frac{{\partial vec(\partial \rho _i^{'}(\psi )/\partial \psi )}}{{\partial \psi {'}}} } \right] } \end{aligned}$$

where

$$\begin{aligned} \frac{{\partial vec(\partial \rho _i^{'} (\psi )/\partial \psi )}}{{\partial \psi '}} = \left[ {\begin{array}{*{20}c} {\frac{{\partial ^2 f(\mathbf{X}_i ,\beta )}}{ {\partial \beta } {\partial \beta ^{'} }}} &{} \mathbf{0} &{} 0 \\ \mathbf{0} &{} \mathbf{0} &{} 0 \\ 0 &{} \mathbf{0} &{} 0 \\ {2f(\mathbf{X}_i ,\beta )\frac{{\partial ^2 f(\mathbf{X}_i ,\beta )}}{{\partial \beta } {\partial \beta ^{'} }}}+{2 \frac{{\partial f(\mathbf{X}_i ,\beta )}}{{\partial \beta }} \frac{{\partial f(\mathbf{X}_i ,\beta )}}{{\partial \beta ^{'} }}} &{} \mathbf{0} &{} 0 \\ \mathbf{0} &{} {\frac{{\partial ^2 \gamma (0)}}{ {\partial \mathbf{a}}{\partial \mathbf{a^{'}}}}} &{} {\frac{{\partial ^2 \gamma (0)}}{{\partial \mathbf{a}} {\partial \sigma ^2 }}} \\ \mathbf{0} &{} {\frac{{\partial ^2 \gamma (0)}}{ {\partial \sigma ^2 } {\partial \mathbf{a}}}} &{} {\frac{{\partial ^2 \gamma (0)}}{ {\partial (\sigma ^2)^2 }}} \\ \end{array}} \right] \nonumber \\ \end{aligned}$$

(7)

Here we obtain $\mathop {\sup }\nolimits _\varGamma \left\| {\frac{{\partial \rho _i^{'}(\psi )}}{{\partial \psi }}{W_i}\frac{{\partial {\rho _i}(\psi )}}{{\partial \psi '}}} \right\| \leqslant \left\| {{W_i}} \right\| \mathop {\sup }\nolimits _\varGamma {\left\| {\frac{{\partial \rho _i^{'}(\psi )}}{{\partial \psi }}} \right\| ^2}$. Using Assumption 5, Cauchy Schwarz inequality, Eq. (6) and similar arguments as in proof of Theorem 2 in Wang and Leblanc (2008) and Theorem 1 above, we can show $\left\| {{W_i}} \right\| \mathop {\sup }\nolimits _\varGamma {\left\| {\frac{{\partial \rho _i^{'}(\psi )}}{{\partial \psi }}} \right\| ^2} \leqslant \left\| {{W_i}} \right\| ^{1+\delta } \mathop {\sup }\nolimits _\varGamma {\left\| {\frac{{\partial \rho _i^{'}(\psi )}}{{\partial \psi }}} \right\| ^{2+\delta } }$ where $E \left\| {{W_i}} \right\| ^{1+\delta } \mathop {\sup }\nolimits _\varGamma {\left\| {\frac{{\partial \rho _i^{'}(\psi )}}{{\partial \psi }}} \right\| ^{2+\delta } < \infty }$, for some $\delta >0$. We also have

$$\begin{aligned}&\mathop {\sup }\limits _\varGamma \left( {\left( {\rho _i^{'}(\psi )\left\| {{W_i}} \right\| \otimes {I_{q+p + 1}}} \right) \left\| {\frac{{\partial {\text {vec(}}\partial \rho _i^{'}(\psi )/\partial \psi )}}{{\partial \psi '}}} \right\| } \right) \\&\quad \leqslant \left( {q+p + 1} \right) \left\| {{W_i}} \right\| \mathop {\sup }\limits _\varGamma \left\| {{\rho _i}(\psi )} \right\| \left\| {\frac{{\partial {\text {vec(}}\partial \rho _i^{'}(\psi )/\partial \psi )}}{{\partial \psi '}}} \right\| \\&\quad \leqslant \left( {q+p + 1} \right) \left[ {{{ \left( {\left\| {{W_i}} \right\| \mathop {\sup }\limits _\varGamma {{\left\| {{\rho _i}(\psi )} \right\| }^2}} \right) }} {{\left( {\left\| {{W_i}} \right\| \mathop {\sup }\limits _\varGamma {{\left\| {\frac{{\partial {\text {vec(}}\partial \rho _i^{'}(\psi )/\partial \psi )}}{{\partial \psi '}}} \right\| }^2}} \right) }}} \right] ^{1/2} \\ \end{aligned}$$

Using Assumption 5, Cauchy Schwarz inequality, Eq. (7) and similar argument as in proof of Theorem 2 in Wang and Leblanc (2008) and Theorem 1 above, we can show for some $\delta >0$

$$\begin{aligned} {{{\left\| {{W_i}} \right\| \mathop {\sup }\limits _\varGamma {{\left\| {\frac{{\partial {\text {vec(}}\partial \rho _i^{'}(\psi )/\partial \psi )}}{{\partial \psi '}}} \right\| }^2}} }} \le {\left\| {W_i} \right\| ^{1+\delta } \mathop {\sup }\limits _\varGamma \left\| {\frac{{\partial vec(\partial \rho _i^{'}(\psi )/\partial \psi )}}{{\partial \psi '}}} \right\| ^{2+\delta } } \end{aligned}$$

where

$$\begin{aligned} E\left( {\left\| {W_i} \right\| ^{1+\delta } \mathop {\sup }\limits _\varGamma \left\| {\frac{{\partial vec(\partial \rho _i^{'}(\psi )/\partial \psi )}}{{\partial \psi '}}} \right\| ^{2+\delta } } \right) <\infty \end{aligned}$$

Define

$$\begin{aligned} \frac{{{\partial ^2}Q(\psi )}}{{\partial \psi \partial \psi {'}}} \!=\! \frac{2}{N}\sum \limits _{i = 1}^N \!{E\left( {\frac{{\partial \rho _i^{'}(\psi )}}{{\partial \psi }}{W_i}\frac{{\partial {\rho _i}(\psi )}}{{\partial \psi }} \!+\! \left( {\rho _i^{'}(\psi ){W_i} \otimes {I_{q+p + 1}}} \right) \frac{{\partial {\text {vec(}}\partial \rho _i^{'}(\psi )/\partial \psi )}}{{\partial \psi '}}} \right) } \end{aligned}$$

It follows from the uniform strong law of large numbers of White and Domowitz (1984) that $\frac{1}{N}\frac{{{\partial ^2}{Q_N}(\psi )}}{{\partial \psi \partial \psi '}}$ and $\frac{{{\partial ^2}Q(\psi )}}{{\partial \psi \partial \psi {'}}}$ converge almost surely and uniformly for all $\psi$ in $\varGamma$ to the same limit. From Lemma 4 of Amemiya (1973), we conclude

$$\begin{aligned} \frac{1}{N}\frac{{{\partial ^2}{Q_N}({{\tilde{\psi }}_N})}}{{\partial \psi \partial \psi '}}\xrightarrow {{a.s.}}\frac{{{\partial ^2}Q({\psi _0})}}{{\partial \psi \partial \psi '}} = 2B \end{aligned}$$

which is due to the fact that

$$\begin{aligned}&E\left[ {\left( {\rho _i^{'}({\psi _0}){W_i} \otimes {I_{q+p + 1}}} \right) \frac{{\partial {\text {vec(}}\partial \rho _i^{'}({\psi _0})/\partial \psi )}}{{\partial \psi '}}} \right] \\&\quad = E\left[ {\left( {E\left( {\rho _i^{'}({\psi _0})|{X_i}} \right) {W_i} \otimes {I_{q+p + 1}}} \right) \frac{{\partial {\text {vec(}}\partial \rho _i^{'}({\psi _0})/\partial \psi )}}{{\partial \psi '}}} \right] = 0 \\ \end{aligned}$$

Since ${\frac{{\partial \rho _i^{'}(\psi _0 )}}{{\partial \psi _0 }}} {W_i}{\rho _i}(\psi _0 )$ are independent with zero mean, by applying Assumption 2 and 5 and Theorem 3.1. in White (1980), it follows as $N \rightarrow \infty$,

$$\begin{aligned} \frac{1}{{\sqrt{N} }}\frac{{\partial {Q_N}({\psi _0})}}{{\partial \psi }}\xrightarrow {L}N(0,4C) \end{aligned}$$

where $C$ is given in Eq. (3). Since $B$ is non singular, for sufficiently large $N$, we have

$$\begin{aligned} \sqrt{N} \left( {{{\hat{\psi }}_{SLSE}} - {\psi _0}} \right) = - {\left( {\frac{1}{N}\frac{{{\partial ^2}{Q_N}({{\tilde{\psi }}_N})}}{{\partial \psi \partial \psi '}}} \right) ^{ - 1}}\frac{1}{{\sqrt{N} }}\frac{{\partial {Q_N}({\psi _0})}}{{\partial \psi }} \end{aligned}$$

Therefore by Assumption 6 and Slutsky’s theorem, we have $\sqrt{N} \left( {{{\hat{\psi }}_{SLSE}} - {\psi _0}} \right) \xrightarrow {L}N(\mathbf{{0}},{B^{ - 1}}C{B^{ - 1}})$.