Large-Scale Gaussian Process Inference with Generalized Histogram Intersection Kernels for Visual Recognition Tasks

Abstract

We present new methods for fast Gaussian process (GP) inference in large-scale scenarios including exact multi-class classification with label regression, hyperparameter optimization, and uncertainty prediction. In contrast to previous approaches, we use a full Gaussian process model without sparse approximation techniques. Our methods are based on exploiting generalized histogram intersection kernels and their fast kernel multiplications. We empirically validate the suitability of our techniques in a wide range of scenarios with tens of thousands of examples. Whereas plain GP models are intractable due to both memory consumption and computation time in these settings, our results show that exact inference can indeed be done efficiently. In consequence, we enable every important piece of the Gaussian process framework—learning, inference, hyperparameter optimization, variance estimation, and online learning—to be used in realistic scenarios with more than a handful of data.

Appendices

Quantization Error Analysis

Proof of Theorem 1

The quantization trick approximates a new example $\mathbf {\varvec{x}}_*$ with a quantized version $\tilde{\mathbf {\varvec{x}}}_*$. In the following, we assume $D$ quantizations with $q$ bins are given for each dimension. For each input value $\mathbf {\varvec{x}}\left( d\right) \in [l_d, u_d ]$, we can compute a quantized value $\tilde{z}$. The maximum quantization error $\varepsilon _{q}$ is then defined as follows:

$$\begin{aligned} \varepsilon _{q}&= \max _{\mathbf {\varvec{x}}\left( d\right) \in [l, u]} \max _{1 \le d \le D} | \mathbf {\varvec{x}}\left( d\right) - \tilde{z} |. \end{aligned}$$

(36)

Our goal is to analyze the difference between the exact predictive mean $\mu _*$ and the one computed with the quantization trick $\tilde{\mu }_*$:

$$\begin{aligned} | \mu _*- \tilde{\mu }_*|&= | \mathbf {\varvec{k}}_{*}^{\mathrm {T}} \mathbf {\varvec{\alpha }} - \tilde{\mathbf {\varvec{k}}}_*^{\mathrm {T}} \mathbf {\varvec{\alpha }} | \nonumber \\&= | (\mathbf {\varvec{k}}_{*}- \tilde{\mathbf {\varvec{k}}}_*)^{\mathrm {T}} \mathbf {\varvec{\alpha }} |\le \sum \limits _{i=1}^N| \triangle k\left( i\right) | | \mathbf {\varvec{\alpha }}\left( i\right) | \end{aligned}$$

(37)

where we have defined $\triangle k\left( i\right) = \kappa (\mathbf {\varvec{x}}_{i}, \mathbf {\varvec{x}}_*) - \kappa (\mathbf {\varvec{x}}_{i}, \tilde{\mathbf {\varvec{x}}}_*)$ as an abbreviation. Let us analyze this term in more detail using the relation $\min (a,b) = \frac{1}{2} \cdot (a + b - |a-b|)$:

$$\begin{aligned} | \triangle k\left( i\right) |&= | \kappa (\mathbf {\varvec{x}}_{i}, \mathbf {\varvec{x}}_*) - \kappa (\mathbf {\varvec{x}}_{i}, \tilde{\mathbf {\varvec{x}}}_*) | \nonumber \\&= \left| \sum \limits _{d=1}^D\min (\mathbf {\varvec{x}}_i\left( d\right) , \mathbf {\varvec{x}}_*\left( d\right) ) - \min (\mathbf {\varvec{x}}_i\left( d\right) , \tilde{\mathbf {\varvec{x}}}_*\left( d\right) ) \right| \nonumber \\&= \frac{1}{2} \left| \sum \limits _{d=1}^D\mathbf {\varvec{x}}_i\left( d\right) + \mathbf {\varvec{x}}_*\left( d\right) - |\mathbf {\varvec{x}}_i\left( d\right) - \mathbf {\varvec{x}}_*\left( d\right) |\right. \ldots \nonumber \\&\quad \left. \ldots -\mathbf {\varvec{x}}_i\left( d\right) - \tilde{\mathbf {\varvec{x}}}_*\left( d\right) + |\mathbf {\varvec{x}}_i\left( d\right) - \tilde{\mathbf {\varvec{x}}}_*\left( d\right) | \right| \nonumber \\&= \frac{1}{2} \left| \sum \limits _{d=1}^D\mathbf {\varvec{x}}_*\left( d\right) - \tilde{\mathbf {\varvec{x}}}_*\left( d\right) - |\mathbf {\varvec{x}}_i\left( d\right) - \mathbf {\varvec{x}}_*\left( d\right) | + \right. \ldots \nonumber \\&\quad \left. \ldots |\mathbf {\varvec{x}}_i\left( d\right) - \tilde{\mathbf {\varvec{x}}}_*\left( d\right) | \right| . \end{aligned}$$

(38)

We can now exploit the fact that $\mathbf {\varvec{x}}_*$ is normalized to sum up to a constant value, e.g., 1. Furthermore, we assume that this also holds for $\tilde{\mathbf {\varvec{x}}}_*$. This assumption is reasonable for a high dimension $D$ and a quantization error with zero mean. Putting both aspects together, we obtain the following equality:

$$\begin{aligned} | \triangle k\left( i\right) |&= \frac{1}{2} \left| \sum \limits _{d=1}^D|\mathbf {\varvec{x}}_i\left( d\right) - \tilde{\mathbf {\varvec{x}}}_*\left( d\right) | - |\mathbf {\varvec{x}}_i\left( d\right) - \mathbf {\varvec{x}}_*\left( d\right) | \right| \nonumber \\&= \frac{1}{2} \left| \Vert \mathbf {\varvec{x}}_{i} - \tilde{\mathbf {\varvec{x}}}_*\Vert _1 - \Vert \mathbf {\varvec{x}}_{i} - \mathbf {\varvec{x}}_*\Vert _1 \right| \end{aligned}$$

(39)

Due to the symmetry in the inequality with respect to $\mathbf {\varvec{x}}_*$ and $\tilde{\mathbf {\varvec{x}}}_*$, we can assume without loss of generality that $\Vert \mathbf {\varvec{x}}_{i} - \tilde{\mathbf {\varvec{x}}}_*\Vert _1 \ge \Vert \mathbf {\varvec{x}}_{i} - \mathbf {\varvec{x}}_*\Vert _1$. This allows us to apply the triangle inequality and we finally obtain:

$$\begin{aligned} | \triangle k\left( i\right) |&= \frac{1}{2} \left( \Vert \mathbf {\varvec{x}}_{i} - \tilde{\mathbf {\varvec{x}}}_*\Vert _1 - \Vert \mathbf {\varvec{x}}_{i} - \mathbf {\varvec{x}}_*\Vert _1 \right) \nonumber \\&\le \frac{1}{2} \left( \Vert \mathbf {\varvec{x}}_{i} - \mathbf {\varvec{x}}_*\Vert _1 + \Vert \mathbf {\varvec{x}}_*- \tilde{\mathbf {\varvec{x}}}_*\Vert _1 - \Vert \mathbf {\varvec{x}}_{i} - \mathbf {\varvec{x}}_*\Vert _1 \right) \nonumber \\&= \frac{1}{2} \Vert \mathbf {\varvec{x}}_*- \tilde{\mathbf {\varvec{x}}}_*\Vert _1. \end{aligned}$$

(40)

This directly leads to $|\triangle k\left( i\right) | \le \frac{D\cdot \varepsilon _{q}}{2}$. Combined with Eq. (37), this proofs the final result of the Theorem.

The bound can be also improved by considering the sum of quantization errors in each dimension, but we skipped this fact for brevity and ease of notation. $\square$

Proof of the Bound

Proof of Lemma 1

First note that due to the conditions of the lemma, the following holds: $1 \le \mu _2 < \mu _1 \le N$ and $\bar{t}> 0$. Furthermore, the bound is only valid for $\beta \ne \bar{t}$, because otherwise the $2 \times 2$ matrix within the bound would be singular.

We now start by deriving the coefficients for $\mu _1$ and $\mu _2$. The first part of Eq. (19) can be written as:

$$\begin{aligned}&\begin{bmatrix} \log \beta ,&\log \bar{t}\end{bmatrix} \begin{bmatrix} \beta&\overline{t} \\ \beta ^2&\overline{t}^2 \end{bmatrix}^{-1} \nonumber \\&\quad = \begin{bmatrix} \log \beta ,&\log \bar{t}\end{bmatrix} \left( \frac{1}{\beta \bar{t}^2 - \bar{t}\beta ^2} \begin{bmatrix} \bar{t}^2&-\bar{t}\\ -\beta ^2&\beta \end{bmatrix} \right) \nonumber \\&\quad = \frac{1}{\beta \bar{t}^2 - \bar{t}\beta ^2} \begin{bmatrix} \bar{t}^2 \log \beta - \beta ^2 \log \bar{t},&\beta \log \bar{t}- \bar{t}\log \beta \end{bmatrix} \nonumber \\&\quad = \frac{1}{\bar{t}- \beta } \begin{bmatrix} \dfrac{\log \beta }{\beta } \bar{t}- \dfrac{\log \bar{t}}{\bar{t}} \beta ,&\dfrac{\log \bar{t}}{\bar{t}} - \dfrac{\log \beta }{\beta } \end{bmatrix}. \end{aligned}$$

(41)

Therefore, we get the following short form of Eq. (19) with $\beta =1$:

(42)

Let $\tilde{\mu }_2$ with $0 < \tilde{\mu }_2 \le \mu _2$ be a lower bound of the squared Frobenius norm. If we replace $\mu _2$ with $\tilde{\mu _2}$ in Eq. (42), we notice that the log-term increases and the denominator of the second part decreases. This directly leads us to the validity of the lemma. $\square$

Proof of Lemma 2

(43)

Now, we show that the second term equals to $N\cdot \log \gamma$ by using the definition of $\bar{t}$ and the calculation of the weights for $\mu _1$ and $\mu _2$ as done in the beginning of the proof of Lemma 1:

(44)

$\square$

Proof of Theorem 2

The first part of the inequality was proved by Bai and Golub (1997) and the proof for the second part is straightforward by applying Lemma 2 with $\gamma = \frac{1}{\beta }$ followed by using Lemma 1:

$\square$

Bounds on Quadratic Forms

From linear algebra we know that any real-valued, symmetric matrix $\varvec{M} \in {\mathbb {R}}^{N\times N}$ can be transformed into $\varvec{M} = \varvec{U} \varvec{D} \varvec{U}^{\mathrm {T}}$ where $\varvec{D}$ is a positive definite diagonal matrix containing the eigenvalues of $\varvec{M}$ and $\varvec{U}$ is an orthogonal matrix of the same size as $\varvec{M}$. Therefore, we notice that for any vector $\mathbf {\varvec{x}}\in {\mathbb {R}}^{N}$ the following holds:

$$\begin{aligned} \mathbf {\varvec{x}}^{\mathrm {T}} \varvec{M} \mathbf {\varvec{x}}= \mathbf {\varvec{x}}^{\mathrm {T}} \varvec{U} \varvec{D} \varvec{U}^{\mathrm {T}} \mathbf {\varvec{x}}= \tilde{\mathbf {\varvec{x}}}^{\mathrm {T}} \varvec{D} \tilde{\mathbf {\varvec{x}}} = \sum _{i = 1}^{N} \lambda _i \tilde{\mathbf {\varvec{x}}} \left( i\right) ^2. \end{aligned}$$

(46)

We denoted with $\lambda _i$ the decreasingly ordered eigenvalues of $\varvec{M}$, i.e., $\lambda _1 \ge \cdots \ge \lambda _{N}$, and $\tilde{x}_i$ contains the projection of $\mathbf {\varvec{x}}$ onto the ith column of $\varvec{U}$, which is the ith eigenvector of $\varvec{M}$. As a result, we can bound the quadratic form in Eq. (46) as follows:

$$\begin{aligned} \mathbf {\varvec{x}}^{\mathrm {T}} \varvec{M} \mathbf {\varvec{x}}&= \sum _{i = 1}^{k} \lambda _i \tilde{\mathbf {\varvec{x}}} \left( i\right) ^2 + \sum _{j = k+1}^{N} \lambda _j \tilde{\mathbf {\varvec{x}}} \left( j\right) ^2 \end{aligned}$$

(47)

$$\begin{aligned}&\le \sum _{i = 1}^{k} \lambda _i \tilde{\mathbf {\varvec{x}}} \left( i\right) ^2 + \lambda _{k+1} \sum _{j = k+1}^{N} \tilde{\mathbf {\varvec{x}}} \left( j\right) ^2. \end{aligned}$$

(48)

Since $\varvec{U}$ is an orthonormal basis, it does not influence the length of vectors, i.e., $\left| \left| \varvec{U} \mathbf {\varvec{x}}\right| \right| = \left| \left| \mathbf {\varvec{x}}\right| \right|$. Therefore, we can obtain the following upper bound:

$$\begin{aligned} \mathbf {\varvec{x}}^{\mathrm {T}} \varvec{M} \mathbf {\varvec{x}}\le \sum _{i = 1}^{k} \lambda _i \tilde{\mathbf {\varvec{x}}} \left( i\right) ^2 + \lambda _{k+1} \Bigl (\left| \left| \mathbf {\varvec{x}}\right| \right| ^2 - \sum _{i = 1}^{k} \tilde{\mathbf {\varvec{x}}} \left( i\right) ^2 \Bigr ). \end{aligned}$$

(49)

Equivalently, we get the following lower bound considering the $k$ smallest eigenvalues of $\varvec{M}$ and bounding the remaining eigenvalues with the $(k+1)$th smallest:

$$\begin{aligned} \begin{aligned} \mathbf {\varvec{x}}^{\mathrm {T}} \varvec{M} \mathbf {\varvec{x}}&\ge \sum _{j = N-k+1}^{N} \lambda _j \tilde{\mathbf {\varvec{x}}} \left( j\right) ^2 \\&\quad + \lambda _{N-k} \Bigl (\left| \left| \mathbf {\varvec{x}}\right| \right| ^2 - \sum _{j = N-k+1}^{N} \tilde{\mathbf {\varvec{x}}} \left( j\right) ^2 \Bigr ). \end{aligned} \end{aligned}$$

(50)

For the special cases of $k= 0$ in Eq. (49) and Eq. (50), we obtain the well known bounds for any positive definite matrix $\varvec{M}$ and any vector $\mathbf {\varvec{x}}$ of corresponding size:

$$\begin{aligned} \lambda _{N}(\varvec{M}) \left| \left| \mathbf {\varvec{x}}\right| \right| ^2 \ \le \ \mathbf {\varvec{x}}^{\mathrm {T}} \varvec{M} \mathbf {\varvec{x}}\ \le \ \lambda _{1}(\varvec{M}) \left| \left| \mathbf {\varvec{x}}\right| \right| ^2. \end{aligned}$$

(51)