Bandwidth matrix selectors for kernel regression

Abstract

Choosing a bandwidth matrix belongs to the class of significant problems in multivariate kernel regression. The problem consists of the fact that a theoretical optimal bandwidth matrix depends on the unknown regression function which to be estimated. Thus data-driven methods should be applied. A method proposed here is based on a relation between asymptotic integrated square bias and asymptotic integrated variance. Statistical properties of this method are also treated. The last two sections are devoted to simulations and an application to real data.

Appendix: Proofs

As the first, we introduce some facts on matrix differential calculus and on the Gaussian density (see Magnus and Neudecker 1979, 1999; Aldershof et al. 1995).

Let $\mathbf {A},\, \mathbf {B}$ be $d\times d$ matrices:

$1^\circ$ :

$\displaystyle {{\mathrm{tr}}}(\mathbf {A}^T \mathbf {B})=\mathrm {vec}^T \mathbf {A}\mathrm {vec}\mathbf {B}$

$2^\circ$ :

$\displaystyle \int \phi _{c\mathbf {I}}(\mathbf {z})\{{{\mathrm{tr}}}(\mathbf {H}^{1/2} D^2 \mathbf {H}^{1/2} \mathbf {z}\mathbf {z}^T)\}m(\mathbf {x})d\mathbf {z}=c\{{{\mathrm{tr}}}(\mathbf {H}D^2)\}m(\mathbf {x})$

$\displaystyle \int \phi _{c\mathbf {I}}(\mathbf {z})\{{{\mathrm{tr}}}^2(\mathbf {H}^{1/2} D^2 \mathbf {H}^{1/2}\mathbf {z}\mathbf {z}^T) \}m(\mathbf {x}) d\mathbf {z}=3c^2\{{{\mathrm{tr}}}^2(\mathbf {H}D^2) \}m(\mathbf {x})$

$\displaystyle \int \phi _{c\mathbf {I}}(\mathbf {z})\{{{\mathrm{tr}}}^k(\mathbf {H}^{1/2} D^2\mathbf {H}^{1/2} \mathbf {z}\mathbf {z}^T) {{\mathrm{tr}}}(\mathbf {H}^{1/2}D\mathbf {z}^T)\}m(\mathbf {x})d\mathbf {z}=\mathbf {0},\quad c,\,k\in \mathbb {N}_0$

$3^\circ$ :

$\displaystyle \int D^km(\mathbf {x})[D^km(\mathbf {x})]^Td\mathbf {x}= (-1)^k\int D^{2k}m(\mathbf {x})m(\mathbf {x})d\mathbf {x},\quad k\in \mathbb {N}$

$4^\circ$ :

$\displaystyle \varLambda (\mathbf {z})=\phi _{4\mathbf {I}}(\mathbf {z})-2\phi _{3\mathbf {I}}(\mathbf {z})+\phi _{2\mathbf {I}}(\mathbf {z})$,

then using $3^\circ$ yields

$\displaystyle \int \varLambda (\mathbf {z})d\mathbf {z}= 0$

$\displaystyle \int \varLambda (\mathbf {z})\{{{\mathrm{tr}}}(\mathbf {H}^{1/2} D^2\mathbf {H}^{1/2} \mathbf {z}\mathbf {z}^T)\}m(\mathbf {x})d\mathbf {z}= \mathbf {0}$

$\displaystyle \int \varLambda (\mathbf {z})\{{{\mathrm{tr}}}^2(\mathbf {H}^{1/2} D^2\mathbf {H}^{1/2} \mathbf {z}\mathbf {z}^T) \}m(\mathbf {x}) d\mathbf {z}=6\{{{\mathrm{tr}}}^2(\mathbf {H}D^2) \}m(\mathbf {x})$

$\displaystyle \int \varLambda (\mathbf {z})\{{{\mathrm{tr}}}^k(\mathbf {H}^{1/2} D^2\mathbf {H}^{1/2} \mathbf {z}\mathbf {z}^T) {{\mathrm{tr}}}(\mathbf {H}^{1/2}D\mathbf {z}^T)\}m(\mathbf {x})d\mathbf {z}=\mathbf {0},\quad k\in \mathbb {N}_0$

$5^\circ$ :

Taylor expansion takes the form

$$\begin{aligned} \begin{aligned} m(\mathbf {x}-\mathbf {H}^{1/2}\mathbf {z})&= m(\mathbf {x})-\{\mathbf {z}^T\mathbf {H}^{1/2}D\}m(\mathbf {x})\\&\quad + \frac{1}{2!}\{(\mathbf {z}^T\mathbf {H}^{1/2}D)^2\}m(\mathbf {x})+\dots \\&\quad +\frac{(-1)^k}{k!}\{(\mathbf {z}^T\mathbf {H}^{1/2}D)^k\}m(\mathbf {x})\\&\quad + o(||\mathbf {H}^{1/2}\mathbf {z}||^k). \end{aligned} \end{aligned}$$

Lemma 2

The integrated square bias can be expressed as

$$\begin{aligned} {{\mathrm{ISB}}}(\mathbf {H}) = \int \left( (K_{\mathbf {H}}*m)(\mathbf {x}) -m(\mathbf {x}) \right) ^2\,{{\mathrm{d}}}\mathbf {x}+ O(n^{-1}), \end{aligned}$$

the symbol $*$ denotes convolution.

Proof

$$\begin{aligned} \widehat{m}(\mathbf {x},\mathbf {H}) =\sum _{i=1}^n \int \limits _{A_i} K_{\mathbf {H}}(\mathbf {x}-\mathbf {z})\,{{\mathrm{d}}}\mathbf {z}Y_i . \end{aligned}$$

Each integral in the sum can be approximated in the following way

$$\begin{aligned} \int \limits _{A_i} K_{\mathbf {H}}(\mathbf {x}-\mathbf {z})\,{{\mathrm{d}}}\mathbf {z}= \underbrace{\lambda (A_i)}_{\frac{1}{n}}K_{\mathbf {H}}(\mathbf {x}-\mathbf {x}_i) + O\left( n^{-1}\right) \end{aligned}$$

Thus

$$\begin{aligned} \widehat{m}(\mathbf {x},\mathbf {H}) =\frac{1}{n}\sum _{i=1}^n K_{\mathbf {H}}(\mathbf {x}-\mathbf {x}_i)Y_i + O\left( n^{-1}\right) . \end{aligned}$$

Further

$$\begin{aligned} E\widehat{m}(\mathbf {x},\mathbf {H})&=\frac{1}{n}\sum _{i=1}^n K_{\mathbf {H}}(\mathbf {x}-\mathbf {x}_i)EY_i + O\left( n^{-1}\right) \\&=\frac{1}{n}\sum _{i=1}^n K_{\mathbf {H}}(\mathbf {x}-\mathbf {x}_i)m(\mathbf {x}_i) + O\left( n^{-1}\right) \\&=\int K_{\mathbf {H}}(\mathbf {x}-\mathbf {z})m(\mathbf {z})d\mathbf {z}+ O\left( n^{-1}\right) \\&= (K_{\mathbf {H}}*m)(\mathbf {x})+ O\left( n^{-1}\right) . \end{aligned}$$

Sketch of the proof of Theorem 1:

Proof

In order to show the $\widehat{\varGamma }$ is the asymptotically unbiased estimator of $\varGamma$, we evaluate $E(\widehat{{{\mathrm{AISB}}}}(\mathbf {H}))$

$$\begin{aligned} \begin{aligned} E(\widehat{{{\mathrm{AISB}}}}(\mathbf {H}))&=\frac{1}{n^2}\sum _{\begin{array}{c} i,j=1 \\ i\ne j \end{array}}^n \varLambda _\mathbf {H}(\mathbf {x}_i-\mathbf {x}_j)m(\mathbf {x}_i)m(\mathbf {x}_j)\\&=\iint \varLambda _\mathbf {H}(\mathbf {x}-\mathbf {y})m(\mathbf {y})m(\mathbf {x})d\mathbf {y}d\mathbf {x}+ O\left( n^{-1}\right) \\&=\iint \varLambda (\mathbf {z})m(\mathbf {x}-\mathbf {H}^{1/2}\mathbf {z})m(\mathbf {x})d\mathbf {z}d\mathbf {x}+ O\left( n^{-1}\right) .\\ \end{aligned} \end{aligned}$$

Taylor expansion, defined in $5^{\circ }$, and using $4^\circ$ yields

$$\begin{aligned} \begin{aligned}&= \iint \varLambda (\mathbf {z})\Biggl (\sum _{i=0}^5\frac{(-1)^i}{i!}\{(\mathbf {z}^T\mathbf {H}^{1/2}D)^i\}m(\mathbf {x}) \\&\phantom {==}+ o(||\mathbf {H}^{1/2}\mathbf {z}||^5)\Biggr )m(\mathbf {x})d\mathbf {z}d\mathbf {x}+ O\left( n^{-1}\right) \\&= \iint \varLambda (\mathbf {z})\biggl (\frac{1}{4!}\{(\mathbf {z}^T\mathbf {H}^{1/2}D)^4\}m(\mathbf {x}) +o(||\mathbf {H}^{1/2}\mathbf {z}||^5)\biggr )m(\mathbf {x})d\mathbf {z}d\mathbf {x}\\&\qquad + O\left( n^{-1}\right) \\&=\frac{1}{4!}\iint \varLambda (\mathbf {z})\{{{\mathrm{tr}}}^2(\mathbf {H}^{1/2} D^2\mathbf {H}^{1/2} \mathbf {z}\mathbf {z}^T) \}m(\mathbf {x})m(\mathbf {x})d\mathbf {z}d\mathbf {x}\\&\qquad +o(||\mathrm {vec}\mathbf {H}||^{5/2})+ O\left( n^{-1}\right) , \\ \end{aligned} \end{aligned}$$

using properties $3^\circ$ and $4^\circ$ we arrive at

$$\begin{aligned} \begin{aligned}&=\frac{1}{4} \int \{{{\mathrm{tr}}}^2(\mathbf {H}D^2) \}m(\mathbf {x}) m(\mathbf {x})d\mathbf {x}+o(||\mathrm {vec}\mathbf {H}||^{5/2}) + O\left( n^{-1}\right) \\&=\frac{1}{4}\int ||\{{{\mathrm{tr}}}(\mathbf {H}D^{2})\}m(\mathbf {x})||^2d\mathbf {x}+o(||\mathrm {vec}\mathbf {H}||^{5/2}) + O\left( n^{-1}\right) \\&=\frac{1}{4}\mathrm {vec}^T\mathbf {H}V(\{\mathrm {vec}D^{2}\}m)\mathrm {vec}\mathbf {H}+o(||\mathrm {vec}\mathbf {H}||^{5/2})+ O\left( n^{-1}\right) . \end{aligned} \end{aligned}$$

To finish the proof of Theorem 1 it is sufficient to derive ${{\mathrm{Var}}}(\widehat{{{\mathrm{AISB}}}}(\mathbf {H}))$

$$\begin{aligned} \begin{aligned} {{\mathrm{Var}}}(\widehat{{{\mathrm{AISB}}}}(\mathbf {H}))&=\frac{1}{n^4}\sum _{\begin{array}{c} i,j=1 \\ i\ne j \end{array}}^n \varLambda ^2_\mathbf {H}(\mathbf {x}_i-\mathbf {x}_j){{\mathrm{Var}}}Y_iY_j\\&=\frac{\sigma ^4}{n^4}\sum _{\begin{array}{c} i,j=1 \\ i\ne j \end{array}}^n \varLambda ^2_\mathbf {H}(\mathbf {x}_i-\mathbf {x}_j). \end{aligned} \end{aligned}$$

Since the estimator $\widehat{{{\mathrm{AISB}}}}(\mathbf {H})$ is asymptotically unbiased and its variance tends to zero the estimator is consistent.

Sketch of the proof of Lemma 1:

Proof

In order to show the $\widehat{\psi }_{4,0}$ is the asymptotically unbiased estimator of $\psi _{4,0}$, we evaluate $E(\widehat{\psi }_{4,0})$

$$\begin{aligned} \begin{aligned} E(\widehat{\psi }_{4,0})&=\frac{1}{n^2}\sum _{\begin{array}{c} i,j=1 \\ i\ne j \end{array}}^n \frac{\partial ^4 K_\mathbf {G}}{\partial x_1^4}(\mathbf {x}_i-\mathbf {x}_j)m(\mathbf {x}_i)m(\mathbf {x}_j)\\&=\iint \frac{\partial ^4 K_\mathbf {G}}{\partial y_1^4}(\mathbf {x}-\mathbf {y})m(\mathbf {y})m(\mathbf {x})d\mathbf {y}d\mathbf {x}+ O\left( n^{-1}\right) \\&=\iint \frac{\partial ^4 m(\mathbf {y})}{\partial y_1^4}K_\mathbf {G}(\mathbf {x}- \mathbf {y})m(\mathbf {x})d\mathbf {y}d\mathbf {x}+ O\left( n^{-1}\right) \\&=\iint \frac{\partial ^4 m(\mathbf {y})}{\partial y_1^4}K(\mathbf {z})m\left( \mathbf {y}+ \mathbf {G}^{1/2}\mathbf {z}\right) d\mathbf {y}d\mathbf {z}+ O\left( n^{-1}\right) \\ \end{aligned} \end{aligned}$$

Taylor expansion defined in $5^{\circ }$ yields

$$\begin{aligned}&= \iint K(\mathbf {z})\Biggl (\sum _{i=0}^2\frac{(-1)^i}{i!}\{(\mathbf {z}^T\mathbf {G}^{1/2}D)^i\}m(\mathbf {y}) \\&\phantom {==}+ o(||\mathbf {G}^{1/2}\mathbf {z}||^2)\Biggr )\frac{\partial ^4 m(\mathbf {y})}{\partial y_1^4}d\mathbf {y}d\mathbf {z}+ O\left( n^{-1}\right) \\&= \int \frac{\partial ^4 m(\mathbf {y})}{\partial y_1^4} m(\mathbf {y})d\mathbf {y}+ \frac{\beta _2}{2}\int \{{{\mathrm{tr}}}(GD^2)\} m(\mathbf {x}) d\mathbf {x}+ O\left( n^{-1}\right) \\&=\,\psi _{4,0} + \frac{\beta _2}{2}\int \{{{\mathrm{tr}}}(GD^2)\} m(\mathbf {x}) d\mathbf {x}+ O\left( n^{-1}\right) . \end{aligned}$$

To finish the proof of Lemma 1 it is sufficient to derive ${{\mathrm{Var}}}(\widehat{\psi }_{4,0})$

$$\begin{aligned} \begin{aligned} {{\mathrm{Var}}}(\widehat{\psi }_{4,0})&=\frac{1}{n^4}\sum _{\begin{array}{c} i,j=1 \\ i\ne j \end{array}}^n \frac{\partial ^4 K_\mathbf {G}}{\partial x_1^4}(\mathbf {x}_i-\mathbf {x}_j){{\mathrm{Var}}}Y_iY_j\\&=\frac{\sigma ^4}{n^4}\sum _{\begin{array}{c} i,j=1 \\ i\ne j \end{array}}^n \frac{\partial ^4 K_\mathbf {G}}{\partial x_1^4}(\mathbf {x}_i-\mathbf {x}_j). \end{aligned} \end{aligned}$$

Since the estimator $\widehat{\psi }_{4,0}$ is asymptotically unbiased and its variance tends to zero, the estimator is consistent. $\square$