Robust relevance vector machine for classification with variational inference

Abstract

The relevance vector machine (RVM) is a widely employed statistical method for classification, which provides probability outputs and a sparse solution. However, the RVM can be very sensitive to outliers far from the decision boundary which discriminates between two classes. In this paper, we propose the robust RVM based on a weighting scheme, which is insensitive to outliers and simultaneously maintains the advantages of the original RVM. Given a prior distribution of weights, weight values are determined in a probabilistic way and computed automatically during training. Our theoretical result indicates that the influences of outliers are bounded through the probabilistic weights. Also, a guideline for determining hyperparameters governing a prior is discussed. The experimental results from synthetic and real data sets show that the proposed method performs consistently better than the RVM if a training data set is contaminated by outliers.

Appendices

Appendix 1: Proof of Proposition 1

The weight value ${{\mathbb {E}}}(w)$ is computed as the mean of $Gamma\left( {w|\tilde{c},\tilde{d}}\right)$, that is

$$\begin{aligned} {{\mathbb {E}}}(w)=\frac{\tilde{c}}{\tilde{d}}=\frac{c}{d-\ln {{\mathbb {E}}}\left( {h\left( {{\varvec{\upbeta }},\xi } \right) }\right) }. \end{aligned}$$

Since the logistic loss function is equivalent to a negative log likelihood function, the weighted logistic loss function can be written as

$$\begin{aligned} {{\mathbb {E}}}(w)l\left\{ {\left( {2t-1}\right) {\varvec{\upbeta }}^{T}{\varvec{\phi }} (\mathbf{x})} \right\}= & {} -{{\mathbb {E}}}(w)\ln p\left( {t|{\varvec{\upbeta }}}\right) \le -{{\mathbb {E}}}(w)\ln {{\mathbb {E}}}\left( {h\left( {{\varvec{\upbeta }},\xi } \right) }\right) \\= & {} -\frac{c\ln {{\mathbb {E}}}\left( {h\left( {{\varvec{\upbeta }},\xi } \right) }\right) }{d-\ln {{\mathbb {E}}}\left( {h\left( {{\varvec{\upbeta }},\xi } \right) }\right) }\le c. \end{aligned}$$

Thus, the weighted logistic loss function is bounded by c.

Appendix 2: Proof of Proposition 2

Recall that $p\left( {t|{\varvec{\upbeta }}}\right) \ge h\left( {{\varvec{\upbeta }},\xi } \right)$. Taking the expectations on both sides with respect to ${\varvec{\upbeta }}$ yields the following result:

$$\begin{aligned} {{\mathbb {E}}}\left( {p\left( {t|{\varvec{\upbeta }}}\right) }\right) =p\left( {t|{\hat{\varvec{{\beta }}}}}\right) \ge {{\mathbb {E}}}\left( {h\left( {{\varvec{\upbeta }},\xi } \right) }\right) \end{aligned}$$

where ${\hat{\varvec{{\beta }}}}$ denotes the expectation of ${\varvec{\upbeta }}$. Since $0\le p\left( {t|{\hat{\varvec{{\beta }}}}}\right) \le 1$, it is always true that ${{\mathbb {E}}}\left( {h\left( {{\varvec{\upbeta }},\xi } \right) }\right) \le 1\Leftrightarrow \ln {{\mathbb {E}}}\left( {h\left( {{\varvec{\upbeta }},\xi } \right) }\right) \le 0$.

$(\Rightarrow )$ If $\ln {{\mathbb {E}}}\left( {h\left( {{\varvec{\upbeta }},\xi _i} \right) }\right)$ is 0, then the weight ${{\mathbb {E}}}(w)$ should be 1 since $\ln {{\mathbb {E}}}\left( {h\left( {{\varvec{\upbeta }},\xi _i} \right) }\right)$ approaches to 0 as $p\left( {t|{\hat{\varvec{{\beta }}}}}\right)$ goes to 1. Therefore, if

$$\begin{aligned} 0\le {{\mathbb {E}}}(w)=\frac{c}{d-\ln {{\mathbb {E}}}\left( {h\left( {{\varvec{\upbeta }},\xi } \right) }\right) }\le 1 \end{aligned}$$

then c should be equal to d.

$(\Leftarrow )$ If $c=d\equiv r$, then

$$\begin{aligned} 0\le {{\mathbb {E}}}(w)=\frac{r}{r-\ln {{\mathbb {E}}}\left( {h\left( {{\varvec{\upbeta }},\xi } \right) }\right) }\le 1 \end{aligned}$$

since $\ln {{\mathbb {E}}}\left( {h\left( {{\varvec{\upbeta }},\xi } \right) }\right)$ is always negative.