Studentized Autoregressive Time Series Residuals

Summary

In this paper we develop large sample approximations for the variances of the residuals obtained from either a least squares or rank analysis of a first order autoregressive process. The formulas are elaborate, but can easily be computed either recursively or via an object oriented language such as S-PLUS. More importantly, our findings indicate that the variances of the residuals depend on much more than just the standard deviation of the error distribution. Thus, we caution against the use of the naive standardization for time series diagnostic procedures. Furthermore, the results can be used to form studentized residuals analogous to those used in the linear regression setting. We compare these new residuals to the conditionally studentized residuals via an example and simulation study. The study reveals some minor differences between the two sets of residuals in regards to outlier detection. Based on our findings, we conclude that the classical studentized linear regression residuals, found in such packages as SAS, SPSS, and RGLM can effectively be used in an autoregressive time series context.

A Lemma and Proofs

The results of the following lemma are used in the proofs of Theorems 2.1 and 2.2.

Lemma A.1 Let ${U_i} = U\left( {{\varepsilon _i}} \right)$, ${\sigma ^2} = E\left[ {\varepsilon _1^2} \right]$, $\sigma _x^2 = E\left[ {X_1^2} \right]$, and for s ≤ t let $\Gamma _s^t = \sum\nolimits_{r = 0}^{t - s} {{\rho ^r}{\varepsilon _{t - r}}}$. When s > t let $\Gamma _s^t = 0$, Furthermore, for some 0 ≤ r ≤ 2, let

$$\begin{array}{*{20}c}{{K_1}} & = & {E\left[ {X_1^{r + 2}} \right]E\left[ {{U_1}} \right] + 2{\rho ^{ - 1}}E\left[ {X_1^{r + 1}} \right]E\left[ {{U_1}{\varepsilon _1}} \right]} \\ {} & + & {{\rho ^{ - 2}}E\left[ {X_1^r} \right]\left( {E\left[ {{U_1}\varepsilon _1^2} \right] - \sigma _x^2E\left[ {{U_1}} \right]} \right)\quad {\rm{ }}and} \\ {{\rm{ }}{K_2}} & = & {E\left[ {X_1^{r + 2}} \right]E\left[ {{U_1}} \right] + {\rho ^{ - 2}}E\left[ {X_1^r} \right]E\left[ {{U_1}} \right]\left( {{\sigma ^2} - \sigma _x^2} \right).} \\ \end{array} $$

Then, model assumptions (1)-(2) imply the following.

• 1. For $s < t,{X_t} = {\rho ^{t - s}}{X_s} + {\rho ^{t - s - 1}}{\varepsilon _{s + 1}} + \Gamma _{s + 2}^t$

• 2. $E\left[ {\Gamma _s^t} \right] = 0$

• 3. $V\left[ {\Gamma _s^t} \right] = \sigma _x^2\left( {1 - {\rho ^{2\left( {t - s + 1} \right)}}} \right)$

• 4. $E\left[ {X_1^2} \right] = \sigma _x^2 = {\sigma ^2}{\left( {1 - {\rho ^2}} \right)^{ - 1}}$

• 5. $E\left[ {X_1^3} \right] = E\left[ {\varepsilon _1^3} \right]{\left( {1 - {\rho ^3}} \right)^{ - 1}}$

• 6. $E\left[ {X_1^4} \right] = \left( {\left( {{{6{\rho ^2}} \over {1 - {\rho ^2}}}} \right){\sigma ^4} + E\left[ {\varepsilon _1^4} \right]} \right){\left( {1 - {\rho ^4}} \right)^{ - 1}}$

• 7. For $\begin{array}{*{20}c}{\quad \quad 1 \le i \le k - 1,E\left[ {X_{i - 1}^r{U_i}{X_{k - 1}}} \right] = {\rho ^{k - i}}E\left[ {X_1^{r + 1}} \right]E\left[ {{U_1}} \right] + } \\ {{\rho ^{k - i - 1}}E\left[ {X_1^r} \right]E\left[ {{U_1}{\varepsilon _1}} \right]} \\ \end{array}$

• 8. For $1 \le i \le k - 1,E\left[ {X_{i - 1}^r{U_i}X_{k - 1}^2} \right] = \sigma _x^2E\left[ {X_1^r} \right]E\left[ {{U_1}} \right] + {\rho ^{2(k - i)}}{K_1}$

• 9. For $k + 1 \le i \le n,E\left[ {X_{k - 1}^r{X_{i - 1}}{U_i}} \right] = {\rho ^{i - k}}E\left[ {X_1^{r + 1}} \right]E\left[ {{U_1}} \right]$

• 10. For $k + 1 \le i \le n,E\left[ {X_{k - 1}^rX_{i - 1}^2{U_i}} \right] = \sigma _x^2E\left[ {X_1^r} \right]E\left[ {{U_1}} \right] + {\rho ^{2(i - k)}}{K_2}$

• 11. For $\begin{array}{*{20}c}{\quad \quad i + 1 \le j \le k - 1{\rm{ }}and{\rm{ }}E\left[ {{U_1}} \right] = 0,E\left[ {{X_{i - 1}}{U_i}{X_{j - 1}}{U_j}X_{k - 1}^2} \right] = } \\ {4{\rho ^{2(k - i)}}{\rho ^{ - 2}}\sigma _x^2{E^2}\left[ {{U_1}{\varepsilon _1}} \right]} \\ \end{array}$

Proof of Lemma A.1. The proof of part 1 follows by exploiting the iterative nature of the AR(1) process. The proof of part 2 follows directly from the fact that E [ε₁] = 0. The proof of part 3 follows from the independence of the errors, the fact that E [ε₁] = 0, and the formula for a finite sum of a geometric series. Parts 4-6 can be proved be letting ${X_1} = \rho {X_0} + {\varepsilon _1}$, expanding the associated polynomial, applying the expectation operator, and then solving for the corresponding expectation. The proofs for the remaining results are similar in nature. Thus, for the sake of brevity, we only present the proof for part 10. To begin, note that the independence of U_i and (X_k−1, X_i−1) implies that $E\left[ {X_{k - 1}^rX_{i - 1}^2{U_i}} \right] = E\left[ {X_{k - 1}^rX_{i - 1}^2} \right]E\left[ {{U_i}} \right]$. Now let ${X_{i - 1}} = {\rho ^{i - k}}{X_{k - 1}} + {\rho ^{i - k - 1}}{\varepsilon _k} + \Gamma _{k + 1}^{i - 1}$, square it, and write $E\left[ {X_{k - 1}^rX_{i - 1}^2} \right]$as a sum of six terms as follows

$$\begin{array}{*{20}c}{E{\rm{ }}\left[ {X_{k - 1}^rX_{i - 1}^2} \right] = {\rho ^{2(i - k)}}E\left[ {X_{k - 1}^{r + 2}} \right] + {\rho ^{2(i - k - 1)}}E\left[ {X_{k - 1}^r} \right]E\left[ {\varepsilon _k^2} \right]} \\ {\quad + \;\;E\left[ {X_{k - 1}^r} \right]V\left[ {\Gamma _{k + 1}^{i - 1}} \right] + 2{\rho ^{2(i - k)}}{\rho ^{ - 1}}E\left[ {X_{k - 1}^{r + 1}} \right]E\left[ {{\varepsilon _k}} \right]} \\ {\quad + \;\;2{\rho ^{i - k}}E\left[ {X_{k - 1}^{r + 1}} \right]E\left[ {\Gamma _{k + 1}^{i - 1}} \right] + 2{\rho ^{i - k - 1}}E\left[ {X_{k - 1}^r} \right]E\left[ {{\varepsilon _k}} \right]E\left[ {\Gamma _{k + 1}^{i - 1}} \right].} \\ \end{array} $$

Part 2 of the lemma and the fact that E [ε₁] = 0 imply that the last three terms are zero. The result now follows from part 3 and direct algebra. □

Proof of Theorem 2.1. Consider the expectation of ${\hat \varepsilon _k}\left( {LS} \right)$. Since E [ε₁] = 0 it follows from (8) that

$$\begin{array}{*{20}c}{E\left[ {{{\hat \varepsilon }_k}(LS)} \right]} & \buildrel\textstyle.\over= & {{{ - 1} \over {n\sigma _x^2}}\sum\limits_{i = 1}^{k - 1} E \left[ {{X_{i - 1}}{\varepsilon _i}{X_{k - 1}}} \right] - {1 \over {n\sigma _x^2}}E\left[ {X_{k - 1}^2{\varepsilon _k}} \right]} \\ {} & - & {{1 \over {n\sigma _x^2}}\sum\limits_{i = k + 1}^n E \left[ {{X_{k - 1}}{X_{i - 1}}{\varepsilon _i}} \right].} \\ \end{array} $$

((18))

Since ε_k is independent of X_k−1 and E [ε₁] = 0 it follows that $E\left[ {X_{k - 1}^2{\varepsilon _k}} \right] = 0$. Furthermore, parts 7 and 9 of Lemma A.l (with r = 1 and U(ε_i) = ε_i) imply that the expectations in the first and third terms of (18) are zero. Thus, $E\left[ {{{\hat \varepsilon }_k}\left( {LS} \right)} \right] \buildrel\textstyle.\over= 0$. Therefore, we need only consider $E\left[ {\hat \varepsilon _k^2\left( {LS} \right)} \right]$ in order to prove the theorem. Direct algebra can be used to show the following

$$\begin{array}{*{20}c}{E\left[ {\hat \varepsilon _k^2(LS)} \right]} & = & {{\sigma ^2} - {2 \over {\sigma _x^2}}{E_1} + {1 \over {\sigma _x^4}}{E_2}{\rm{ where}}} \\ {{E_1}} & = & {{1 \over n}\sum\limits_{i = 1}^n E \left[ {{X_{i - 1}}{\varepsilon _i}{X_{k - 1}}{\varepsilon _k}} \right]{\rm{ and}}} \\ {{E_2}} & = & {{1 \over {{n^2}}}\sum\limits_{i = 1}^n E \left[ {X_{i - 1}^2\varepsilon _i^2X_{k - 1}^2} \right] + {2 \over {{n^2}}}\sum\limits_{i < j} E \left[ {{X_{i - 1}}{\varepsilon _i}{X_{j - 1}}{\varepsilon _j}X_{k - 1}^2} \right]} \\ {} & = & {{E_{21}} + {E_{22}}{\rm{ say}}{\rm{.}}} \\ \end{array} $$

Consider the expectation in E₁ first. For i ≠ k either ε_i or ε_k is independent of the remaining terms. This implies that these n − 1 expectations are zero since E[ε₁] = 0. Hence

$${E_1} = {1 \over n}E\left[ {X_{k - 1}^2\varepsilon _k^2} \right] = {1 \over n}\sigma _x^2{\sigma ^2}.$$

((19))

Next, rewrite E₂₁ as follows

$${E_{21}} = {1 \over n}\sum\limits_{i = 1}^{k - 1} E \left[ {X_{i - 1}^2\varepsilon _i^2X_{k - 1}^2} \right] + {1 \over n}E\left[ {X_{k - 1}^4\varepsilon _k^2} \right] + {1 \over n}\sum\limits_{i = k + 1}^n E \left[ {X_{k - 1}^2X_{i - 1}^2\varepsilon _i^2} \right].$$

Part 8 of Lemma A.1 (with r = 2 and $\left. {U\left( {{\varepsilon _i}} \right) = \varepsilon _i^2} \right)$ can be used to calculate the expectation in the first term of E₂₁. That is

$$\begin{array}{*{20}c}{E\left[ {X_{i - 1}^2\varepsilon _i^2X_{k - 1}^2} \right] = \sigma _x^4{\sigma ^2} + {\rho ^{2\left( {k - i} \right)}}{K_1}\left( {LS} \right)\quad {\rm{where}}} \\ {{K_1}\left( {LS} \right) = E\left[ {X_1^4} \right]{\sigma ^2} + 2{\rho ^{ - 1}}E\left[ {X_1^3} \right]E\left[ {\varepsilon _1^3} \right] + {\rho ^{ - 2}}\sigma _x^2\left( {E\left[ {\varepsilon _1^4} \right] - \sigma _x^2{\sigma ^2}} \right).} \\ \end{array} $$

((20))

The second term in E₂₁ is $\left( {1/n} \right)E\left[ {X_1^4} \right]{\sigma ^2}$. Finally, part 10 of Lemma A.l (with r = 2 and $U\left( {{\varepsilon _i}} \right) = \varepsilon _i^2$) can be used to calculate the expectation in the last term of E₂₁. That is

$$\begin{array}{*{20}c}{E\left[ {X_{k - 1}^2X_{i - 1}^2\varepsilon _i^2} \right] = \sigma _x^4{\sigma ^2} + {\rho ^{2\left( {i - k} \right)}}{K_2}\left( {LS} \right)\quad {\rm{where}}} \\ {{K_2}\left( {LS} \right) = E\left[ {X_1^4} \right]{\sigma ^2} + {\rho ^{ - 2}}\sigma _x^2{\sigma ^2}\left( {{\sigma ^2} - \sigma _x^2} \right).} \\ \end{array} $$

((21))

It follows from (20), (21), and the formula for a finite sum of a geometric series that

$$\begin{array}{*{20}c}{{E_{21}}} & = & {{{n - 1} \over {{n^2}}}\left( {\sigma _x^4{\sigma ^2}} \right) + {{E\left[ {X_1^4} \right]{\sigma ^2}} \over {{n^2}}} + {{{K_1}(LS)} \over {{n^2}}}\sum\limits_{j = 1}^{k - 1} {{\rho ^{2j}}} + {{{K_2}(LS)} \over {{n^2}}}\sum\limits_{j = 1}^{n - k} {{\rho ^{2j}}} } \\ {} & = & {{{n - 1} \over {{n^2}}}\left( {\sigma _x^4{\sigma ^2}} \right) + {{E\left[ {X_1^4} \right]{\sigma ^2}} \over {{n^2}}} + {{{K_1}(LS)} \over {{n^2}}}\left( {{{{\rho ^2} - {\rho ^{2k}}} \over {1 - {\rho ^2}}}} \right)} \\ {} & + & {{{{K_2}(LS)} \over {{n^2}}}\left( {{{{\rho ^2} - {\rho ^{2(n - k + 1)}}} \over {1 - {\rho ^2}}}} \right).} \\ \end{array} $$

((22))

Lastly, consider E₂₂ and note that there are basically five cases: k < i < j, k = i < j, i < k < j, i < j = k, and i < j < k. For the first four cases note that ε_j is independent from the remaining terms given in the corresponding expectation. Since E[ε₁] = 0, this implies that E₂₂ = 0 for the first four cases. For the last case use part 11 of Lemma A.l (with U(ε_i) = ε_i) to show the following

$$\begin{array}{*{20}c}{{E_{22}}} & = & {{2 \over {{n^2}}}\sum\limits_{i = 1}^{k - 2} {\sum\limits_{j = i + 1}^{k - 1} E } \left[ {{X_{i - 1}}{\varepsilon _i}{X_{j - 1}}{\varepsilon _j}X_{k - 1}^2} \right]} \\ {} & = & {{{8\sigma _x^2{\sigma ^4}} \over {{n^2}{\rho ^2}}}\sum\limits_{i = 1}^{k - 2} {(k - i - 1)} {\rho ^{2(k - i)}}} \\ {} & = & {{{8\sigma _x^2{\sigma ^4}} \over {{n^2}}}\sum\limits_{i = 1}^{k - 2} i {\rho ^{2i}}} \\ {} & = & {{{8\sigma _x^2{\sigma ^4}{\rho ^2}} \over {{n^2}}}\left[ {{{1 - (k - 1){\rho ^{2(k - 2)}} + (k - 2){\rho ^{2(k - 1)}}} \over {{{\left( {1 - {\rho ^2}} \right)}^2}}}} \right].} \\ \end{array} $$

((23))

Combining (19), (22), and (23) along with the expectations given in parts 4-6 of Lemma A.l completes the proof. □