Minimizing variable selection criteria by Markov chain Monte Carlo

Abstract

Regression models with a large number of predictors arise in diverse fields of social sciences and natural sciences. For proper interpretation, we often would like to identify a smaller subset of the variables that shows the strongest information. In such a large size of candidate predictors setting, one would encounter a computationally cumbersome search in practice by optimizing some criteria for selecting variables, such as AIC, $C_{P}$ and BIC, through all possible subsets. In this paper, we present two efficient optimization algorithms vis Markov chain Monte Carlo (MCMC) approach for searching the global optimal subset. Simulated examples as well as one real data set exhibit that our proposed MCMC algorithms did find better solutions than other popular search methods in terms of minimizing a given criterion.

Appendices

Appendix 1: Proofs

Proof of Theorem 1

Assume $Q_{q,i}$, $Q_{q,j}\in \mathcal {Q}_{q}$. Let $E_{i,j}=Q_{q,i}\cap Q_{q,j}$ and $n_{i,j}=q-|E_{i,j}|$. According to MCMC1 algorithm, we have

$$\begin{aligned} p_{ij}=\left\{ \begin{array}{lll} \sum _{\eta \in Q_{q,i}}\frac{1}{q}\cdot \frac{1}{\tilde{z}_{Q_{q,i}\setminus \{\eta \}}}\cdot f(\text {RSS}_{Q_{q,i}}) &{}\quad \text{ if } n_{i,j}= 0, \\ \frac{1}{q}\cdot \frac{1}{\tilde{z}_{E_{i,j}}}\cdot f(\text {RSS}_{Q_{q,j}}) &{}\quad \text{ if } n_{i,j}= 1, \\ 0 &{}\quad \text{ otherwise, } \end{array} \right. \end{aligned}$$

where $\tilde{z}_{B}=\sum _{\kappa \notin B} f(\text {RSS}_{B\cup \{\kappa \}})$, $B\in \mathcal {Q}_{q-1}$. For the convergence, by Ergodic Theorem (see “Appendix 2”), it suffices to show that there exists an $n_{0}$ such that $p_{ij}^{(n_{0})}>0$ for all i, j. First, consider the case $j=i$,

$$\begin{aligned} p_{ii}^{(q)}&\ge \prod _{k=1}^{q} \Pr \left( x_{k}=Q_{q,i}\mid x_{k-1}=Q_{q,i}\right) \nonumber \\&= p^{q}_{ii} \nonumber \\&= \left[ \sum _{\eta \in Q_{q,i}}\frac{1}{q}\cdot \frac{1}{\tilde{z}_{Q_{q,i}\setminus \{\eta \}}} \cdot f(\text {RSS}_{Q_{q,i}})\right] ^{q} ~~~>0.\nonumber \end{aligned}$$

For $i \ne j$, let

$$\begin{aligned} Q^{*}_{q,i}=Q_{q,i}{\setminus }E_{i,j},~~ Q^{*}_{q,j}=Q_{q,j}{\setminus }E_{i,j}, \end{aligned}$$

and define

$$\begin{aligned} Q^{*(k)} = Q^{*}_{q,i}(k+1:n_{i,j}) \cup E_{i,j} \cup Q^{*}_{q,j}(1:k) \quad \quad \quad 0\le k \le n_{i,j}, \end{aligned}$$

where $Q^{*}_{q,j}(1:k)$ denotes the first k variables in $Q^{*}_{q,j}$ and $Q^{*}_{q,i}(k+1:n_{i,i})$ denotes the set $Q^{*}_{q,i}$ excluding the first k variables. In this setup, for $1\le k\le n_{i,j}$, $q-|Q^{*(k-1)} \cap Q^{*(k)}|=1$, which leads to

$$\begin{aligned} \Pr \left( x_{k}=Q^{*(k)}\mid x_{k-1}=Q^{*(k-1)}\right) >0. \end{aligned}$$

Therefore,

$$\begin{aligned} p_{ij}^{(q)}&=\sum _{\alpha }p^{(n_{i,j})}_{i\alpha }p^{(q-n_{i,j})}_{\alpha j} \nonumber \\&\ge p^{(n_{i,j})}_{i j}p^{(q-n_{i,j})}_{jj} \nonumber \\&\ge \prod _{k=1}^{n_{i,j}} \Pr \left( x_{k}=Q^{*(k)}\mid x_{k-1}=Q^{*(k-1)}\right) \cdot p^{q-n_{i,j}}_{jj} >0,\nonumber \end{aligned}$$

where $p^{(0)}_{\alpha j}:=1$ for all $\alpha$. Since $p_{ij}^{(q)}>0$ for all i, j, $p_{ij}^{(m)}$’s converge by Ergodic Theorem. Next, we will show that $p_{ij}^{(m)}$ converges to $f(\text {RSS}_{Q_{q,j}})/z$ for $j=1,\ldots ,C^{p}_{q}$. For $n_{i,j}\ne 1$, it is clear that

$$\begin{aligned} \frac{1}{z} \cdot f(\text {RSS}_{Q_{q,i}}) \cdot p_{ij} =\frac{1}{z}\cdot f(\text {RSS}_{Q_{q,j}})\cdot p_{ji}. \end{aligned}$$

For $n_{i,j}=1$,

$$\begin{aligned} \frac{1}{z} \cdot f(\text {RSS}_{Q_{q,i}}) \cdot p_{ij}&= \frac{1}{z} \cdot f(\text {RSS}_{Q_{q,i}}) \cdot \frac{1}{q}\cdot \frac{1}{\tilde{z}_{E_{i,j}}}\cdot f(\text {RSS}_{Q_{q,j}})\nonumber \\&= \frac{1}{z} \cdot f(\text {RSS}_{Q_{q,j}}) \cdot \frac{1}{q}\cdot \frac{1}{\tilde{z}_{E_{j,i}}}\cdot f(\text {RSS}_{Q_{q,i}})\nonumber \\&= \frac{1}{z} \cdot f(\text {RSS}_{Q_{q,j}}) \cdot p_{ji}.\nonumber \end{aligned}$$

By Remark 3 (see “Appendix 2”) , we prove $\pi _{j}=f(\text {RSS}_{Q_{q,j}})/z$, $j=1,\ldots ,C^{p}_{q}$.   $\square$

Proof of Theorem 2

The proof is very similar to that of MCMC1. Assume $Q_{q,i}$, $Q_{q,j}\in \mathcal {Q}_{q}$. Let $E_{i,j}=Q_{q,i}\cap Q_{q,j}$ and $n_{i,j}=q-|E_{i,j}|$. According to MCMC2 algorithm, we have

$$\begin{aligned} p_{ij}=\left\{ \begin{array}{lll} \sum _{\eta \in Q_{q,i}} \frac{1}{z_{Q_{q,i}}} \cdot f(\text {RSS}_{Q_{q,i}\setminus \{\eta \}}) \cdot \frac{1}{\tilde{z}_{Q_{q,i}\setminus \{\eta \}}}\cdot f(\text {RSS}_{Q_{q,i}}) &{}\qquad \text{ if } n_{i,j}= 0, \\ \frac{1}{z_{Q_{q,i}}} \cdot f(\text {RSS}_{E_{i,j}})\cdot \frac{1}{\tilde{z}_{E_{i,j}}}\cdot f(\text {RSS}_{Q_{q,j}}) &{}\qquad \text{ if } n_{i,j}= 1, \\ 0&{} \qquad \text{ otherwise, } \end{array} \right. \end{aligned}$$

where $z_{A}=\sum _{\zeta \in A}f(\text {RSS}_{A\setminus \{\zeta \}})$ for $A\in \mathcal {Q}_{q}$, and $\tilde{z}_{B}=\sum _{\kappa \notin B} f(\text {RSS}_{B\cup \{\kappa \}})$ for $B\in \mathcal {Q}_{q-1}$. To apply Ergodic Theorem, we claim that $p_{ij}^{(q)}>0$ in the following. For $j=i$,

$$\begin{aligned} p_{ii}^{(q)} \ge \left[ \sum _{\eta \in Q_{q,i}} \frac{1}{z_{Q_{q,i}}} \cdot f(\text {RSS}_{Q_{q,i}\setminus \{\eta \}}) \cdot \frac{1}{\tilde{z}_{Q_{q,i}\setminus \{\eta \}}}\cdot f(\text {RSS}_{Q_{q,i}})\right] ^{q} ~~~>0. \end{aligned}$$

For $i\ne j$, following the same arguments and notations of the proof of Theorem 1, we have

$$\begin{aligned} p_{ij}^{(q)} \ge \prod _{k=1}^{n_{i,j}} \Pr \left( x_{k}=Q^{*(k)}\mid x_{k-1}=Q^{*(k-1)}\right) \cdot p^{q-n_{i,j}}_{jj} >0. \end{aligned}$$

Since $p_{ij}^{(q)}>0$ for all i, j, $p_{ij}^{(m)}$’s converge by Ergodic Theorem. Next, we will show that $p_{ij}^{(m)}$ converges to $\frac{1}{z} \cdot \sum _{\zeta \in Q_{q,j}}f(\text {RSS}_{Q_{q,j}\setminus \{\zeta \}}) \cdot f(\text {RSS}_{Q_{q,j}})$ for $j=1,\ldots ,C^{p}_{q}$. For $n_{i,j}\ne 1$, it is trivial that

$$\begin{aligned} \frac{z_{Q_{q,i}}}{z}\cdot f(\text {RSS}_{Q_{q,i}})\cdot p_{ij} =\frac{z_{Q_{q,j}}}{z}\cdot f(\text {RSS}_{Q_{q,j}})\cdot p_{ji}. \end{aligned}$$

For $n_{i,j}=1$,

$$\begin{aligned}&\frac{z_{Q_{q,i}}}{z} \cdot f(\text {RSS}_{Q_{q,i}}) \cdot p_{ij} \nonumber \\&\quad = \frac{z_{Q_{q,i}}}{z} \cdot f(\text {RSS}_{Q_{q,i}}) \cdot \frac{1}{z_{Q_{q,i}}} \cdot f(\text {RSS}_{E_{i,j}})\cdot \frac{1}{\tilde{z}_{E_{i,j}}}\cdot f(\text {RSS}_{Q_{q,j}})\nonumber \\&\quad = \frac{z_{Q_{q,j}}}{z} \cdot f(\text {RSS}_{Q_{q,j}}) \cdot \frac{1}{z_{Q_{q,j}}} \cdot f(\text {RSS}_{E_{j,i}})\cdot \frac{1}{\tilde{z}_{E_{j,i}}}\cdot f(\text {RSS}_{Q_{q,i}})\nonumber \\&\quad = \frac{z_{Q_{q,j}}}{z} \cdot f(\text {RSS}_{Q_{q,j}}) \cdot p_{ji}.\nonumber \end{aligned}$$

By Remark 3 (see “Appendix 2”), we have

$$\begin{aligned} \pi _{j}&= \frac{z_{Q_{q,j}}}{z} \cdot f(\text {RSS}_{Q_{q,j}}) \nonumber \\&= \frac{1}{z} \cdot \sum _{\zeta \in Q_{q,j}}f(\text {RSS}_{Q_{q,j}\setminus \{\zeta \}}) \cdot f(\text {RSS}_{Q_{q,j}})\,\,\,\,j=1,\ldots ,C^{p}_{q}. \nonumber \end{aligned}$$

$\square$

Appendix 2

Ergodic Theorem

Let $\varGamma =[p_{ij}]$ be the transition matrix of a chain with a finite state space $\mathbf S =\{1, 2, \ldots , S\}$. If there is an $n_{0}$ such that

$$\begin{aligned} \min _{i,\,j}p_{ij}^{(n_{0})}>0, \end{aligned}$$

then there is a sequence of numbers $\pi _{1}, \ldots , \pi _{S}$ such that

$$\begin{aligned} \pi _{j}>0,\,\,\,\,\sum _{j}\pi _{j}=1 \end{aligned}$$

(5)

and

$$\begin{aligned} p_{ij}^{(m)}\rightarrow \pi _{j},\,\,\,\,m\rightarrow \infty \end{aligned}$$

for every $i\in \mathbf S$. Moreover, the sequence $\pi _{1}, \ldots , \pi _{S}$ is the only stationary probability distribution with transition matrix $\varGamma$, that is, the set $\pi _{1}, \ldots , \pi _{S}$ is the only one which satisfies both (5) and the following equations

$$\begin{aligned} \pi _{j}=\sum _{\alpha }\pi _{\alpha }p_{\alpha j},\,\,\,\,j=1,\ldots ,S. \nonumber \end{aligned}$$

Proof

Vide Shiryaev (1996, p. 118–120). $\square$

Remark 3

If there is a sequence of numbers $\tilde{\pi }_{1}, \ldots , \tilde{\pi }_{S}$ which satisfies (5) and the equations

$$\begin{aligned} \tilde{\pi }_{i} p_{ij}=\tilde{\pi }_{j}p_{ji},\,\,\,\,i,j=1,\ldots ,S, \nonumber \end{aligned}$$

then $\tilde{\pi }_{i}=\pi _{i}$ for $i=1,\ldots ,S$.