Stein Variational Gradient Descent with Multiple Kernels

Abstract

Bayesian inference is an important research area in cognitive computation due to its ability to reason under uncertainty in machine learning. As a representative algorithm, Stein variational gradient descent (SVGD) and its variants have shown promising successes in approximate inference for complex distributions. In practice, we notice that the kernel used in SVGD-based methods has a decisive effect on the empirical performance. Radial basis function (RBF) kernel with median heuristics is a common choice in previous approaches, but unfortunately, this has proven to be sub-optimal. Inspired by the paradigm of Multiple Kernel Learning (MKL), our solution to this flaw is using a combination of multiple kernels to approximate the optimal kernel, rather than a single one which may limit the performance and flexibility. Specifically, we first extend Kernelized Stein Discrepancy (KSD) to its multiple kernels view called Multiple Kernelized Stein Discrepancy (MKSD) and then leverage MKSD to construct a general algorithm Multiple Kernel SVGD (MK-SVGD). Further, MK-SVGD can automatically assign a weight to each kernel without any other parameters, which means that our method not only gets rid of optimal kernel dependence but also maintains computational efficiency. Experiments on various tasks and models demonstrate that our proposed method consistently matches or outperforms the competing methods.

Appendices

Appendix A. Definitions

Definition 2

(Strictly Positive Kernel) For any function f that satisfies $0 \le \Vert f\Vert _2^2 \le \infty$, a kernel $k(x,x')$ is said to be integrally strictly positive definition if

$$\begin{aligned} \int _{\mathcal {X}} f(x)k(x,x')f(x') dx dx' > 0. \end{aligned}$$

Definition 3

(Stein Class) For a smooth function $f: \mathcal {X} \rightarrow \mathbb {R}$, then, we can say that f is in the Stein class of q if satisfies

$$\int _{x \in \mathcal {X}} \nabla _{x}(f(x) q(x)) d x=0$$

Definition 4

(Kernel in Stein Class) If kernel $k\left( x, x^{\prime }\right)$ has continuous second-order partial derivatives, and for any fixed x, both $k(x, \cdot )$ and $k(\cdot , x)$ are in the Stein class of p, then the kernel $k\left( x, x^{\prime }\right)$ said to be in the Stein class of p.

Appendix B. Proof

Proof of Proposition 1

Proof

For the kernel $k_{\textbf {w}}(x,x')=\sum _{i=1}^m w_i k_i(x,x')$, where $\textbf {w} \in \mathbb {R}_{+}^m, ||\textbf {w}||_{2}=1$, and $k_i(x,x')$ is in the Stein class. According to Definition 4, we know that $k_i(x,x')$ has continuous second-order partial derivatives. Setting $g_i$ be the continuous second-order partial, we know that the continuous second-order partial g of multiple kernel $k_{\textbf {w}}$ is

$$\begin{aligned} g = \sum _{i=1}^{m} w_i g_i \nonumber \end{aligned}$$

So, the kernel $k_{\textbf {w}}(x,x')=\sum _{i=1}^m w_i k_i(x,x')$, where $\textbf {w} \in \mathbb {R}_{+}^m, ||\textbf {w}||_{2}=1$, is in the Stein class.

Proof of Proposition 2

Proof

We know $\mathbb {S}_{k_i}(q\Vert p) \ge 0$ and $\mathbb {S}_{k_i}(q\Vert p) = 0$ if and only if $q=p$ a.e. This means $\mathbb {S}_{k_{\textbf {w}}} \ge 0$ and $\mathbb {S}_{k_{\textbf {w}}}(q,p) = 0$ when $q = p$ a.e. for any $\textbf {w} \in \mathbb {R}_{+}^{m}$. Then, we know $\mathbb {S}_{k_{\textbf {w}}}(q,p) = 0$ if and only if $\mathbb {S}_{k_i}(q,p) = 0$. Thus $q = p$ a.e.

Appendix C. Comparison to Matrix-SVGD (average)

Although using a similar “mixture” form, our approach is intrinsically different from Matrix-SVGD (average). Next, we will detailedly compare the two methods from theoretical aspect.

To elaborate more clearly, we briefly review the “mixture preconditioning kernel” in the Matrix-SVGD, which has the form of:

$$\begin{aligned} \varvec{K}\left( \varvec{x}, \varvec{x}^{\prime }\right) =\sum _{\ell =1}^{m} \varvec{K}_{\varvec{Q}_{\ell }}\left( \varvec{x}, \varvec{x}^{\prime }\right) w_{\ell }(\varvec{x}) w_{\ell }\left( \varvec{x}^{\prime }\right) \end{aligned}$$

(25)

where $\varvec{K}_{\varvec{Q}_{\ell }}\left( \varvec{x}, \varvec{x}^{\prime }\right)$ is defined as

$$\begin{aligned} \varvec{K}_{\varvec{Q}_{\ell }}\left( \varvec{x}, \varvec{x}^{\prime }\right) :=\varvec{Q}_{\ell }^{-1 / 2} \varvec{K}_{0}\left( \varvec{Q}_{\ell }^{1 / 2} \varvec{x}, \varvec{Q}_{\ell }^{1 / 2} \varvec{x}^{\prime }\right) \varvec{Q}_{\ell }^{-1 / 2}, \end{aligned}$$

(26)

and $w_{\ell }(\varvec{x})$ is defined as

$$\begin{aligned} w_{\ell }(\varvec{x})=\frac{\mathcal {N}\left( \varvec{x} ; \varvec{z}_{\ell }, \varvec{Q}_{\ell }^{-1}\right) }{\sum _{\ell ^{\prime }=1}^{m} \mathcal {N}\left( \varvec{x} ; \varvec{z}_{\ell ^{\prime }}, \varvec{Q}_{\ell ^{\prime }}^{-1}\right) }. \end{aligned}$$

(27)

$\varvec{K}_{0}$ can be chosen to be the standard RBF kernel and $\mathcal {N}$ is Gaussian distribution.

The kernel used in our method is defined as

$$\begin{aligned} k_{\textbf {w}}(\varvec{x},\varvec{x}')=\sum _{i}^{m} w_i k_i(\varvec{x},\varvec{x}'), \quad \text {s.t.} \quad \textbf {w} \in \mathbb {R}_{+}^m, ||\textbf {w}||_{2}=1 \end{aligned}$$

(28)

as shown in Eq. (13) in “ Multiple Kernelized Stein Discrepancy’’. Therefore, we can summarize the difference between the two as follows.

• Different RKHS. The Matrix-SVGD introduces the optimization problem to vector-valued RKHS with matrix-valued kernels, while our method focuses on leveraging the effectiveness of Multiple Kernel Learning, which optimizes the objective in scalar-valued RKHS. So, the value of kernel $\varvec{K}_{\varvec{Q}_{\ell }}\left( \varvec{x}, \varvec{x}^{\prime }\right)$ in Eq. (25) is a matrix while our kernel is still a scalar. The resultant value of the mixture kernel is also different: vector-value for Matrix-SVGD and scalar-value for ours.

• Different roles. In Matrix-SVGD(average), the “mixture” form was introduced to address the intractability of the “Point-wise Preconditioning” matrix, which is done by using a weighted combination of several constant preconditioning matrices associated with a set of anchor points. In contrast, taking inspiration from the paradigm of multiple kernel learning, we utilize a more powerful “mixture” kernel instead of a single one to approximate the optimal kernel.

• Different weight’s updating way. According to Eq. (27), we know that the kernel’s weight is the product of two Gaussian mixture probability from the anchor points, in which the weight’s distribution is pre-defined. While in our method, the optimal weight of each kernel is learned automatically, as shown in Eq. (21) in “SVGD with Multiple Kernels’’.