Localized algorithms for multiple kernel learning

Abstract

Instead of selecting a single kernel, multiple kernel learning (MKL) uses a weighted sum of kernels where the weight of each kernel is optimized during training. Such methods assign the same weight to a kernel over the whole input space, and we discuss localized multiple kernel learning (LMKL) that is composed of a kernel-based learning algorithm and a parametric gating model to assign local weights to kernel functions. These two components are trained in a coupled manner using a two-step alternating optimization algorithm. Empirical results on benchmark classification and regression data sets validate the applicability of our approach. We see that LMKL achieves higher accuracy compared with canonical MKL on classification problems with different feature representations. LMKL can also identify the relevant parts of images using the gating model as a saliency detector in image recognition problems. In regression tasks, LMKL improves the performance significantly or reduces the model complexity by storing significantly fewer support vectors.

Introduction

Support vector machine (SVM) is a discriminative classifier based on the theory of structural risk minimization [33]. Given a sample of independent and identically distributed training instances $\{({\mathit{x}}_i,y_i){\}}_{i=1}^N$, where ${\mathit{x}}_i\in {\mathbb{R}}^D$ and $y_i\in \{-1,+1\}$ is its class label, SVM finds the linear discriminant with the maximum margin in the feature space induced by the mapping function $\Phi (·)$. The discriminant function is

$$f(\mathit{x})=〈\mathit{w},\Phi (\mathit{x})〉+b$$

whose parameters can be learned by solving the following quadratic optimization problem:

$$\mathrm{min.}{\displaystyle \frac{1}{2}}\parallel \mathit{w}{\parallel }_2^2+C\sum _{i=1}^N{\xi }_i\mathrm{w.r.t.}\mathit{w}\in {\mathbb{R}}^S,\mathit{\xi }\in {\mathbb{R}}_{+}^N,b\in \mathbb{R}\mathrm{s.t.}{y}_i(〈\mathit{w},\Phi ({\mathit{x}}_i)〉+b)\ge 1-{\xi }_i\forall i$$

where $\mathit{w}$ is the vector of weight coefficients, S is the dimensionality of the feature space obtained by $\Phi (·)$, C is a predefined positive trade-off parameter between model simplicity and classification error, $\mathit{\xi }$ is the vector of slack variables, and b is the bias term of the separating hyperplane. Instead of solving this optimization problem directly, the Lagrangian dual function enables us to obtain the following dual formulation:

$$\mathrm{max.}\sum _{i=1}^N{\alpha }_i-{\displaystyle \frac{1}{2}}\sum _{i=1}^N\sum _{j=1}^N{\alpha }_i{\alpha }_i{y}_i{y}_{j}k({\mathit{x}}_i,{\mathit{x}}_j)\mathrm{w.r.t.}\mathit{\alpha }\in [0,C{]}^N\mathrm{s.t.}\sum _{i=1}^N{\alpha }_i{y}_i=0$$

where $\mathit{\alpha }$ is the vector of dual variables corresponding to each separation constraint and the obtained kernel matrix of $k({\mathit{x}}_i,{\mathit{x}}_j)=〈\Phi ({\mathit{x}}_i),\Phi ({\mathit{x}}_j)〉$ is positive semidefinite. Solving this, we get $\mathit{w}={\sum }_{i=1}^N{\alpha }_i{y}_i\Phi ({\mathit{x}}_i)$ and the discriminant function can be written as

$$f(\mathit{x})=\sum _{i=1}^N{\alpha }_i{y}_{i}k({\mathit{x}}_i,\mathit{x})+b.$$

There are several kernel functions successfully used in the literature such as the linear kernel (k_L), the polynomial kernel (k_P), and the Gaussian kernel (k_G)

$$k_L({\mathit{x}}_i,{\mathit{x}}_j)=〈{\mathit{x}}_i,{\mathit{x}}_j〉k_P({\mathit{x}}_i,{\mathit{x}}_j)=(〈{\mathit{x}}_i,{\mathit{x}}_j〉+1{)}^{q}q\in \mathbb{N}{k}_G({\mathit{x}}_i,{\mathit{x}}_j)=\exp (-\parallel {\mathit{x}}_i-{\mathit{x}}_j{\parallel }_2^2/s^2)s\in {\mathbb{R}}_{++}.$$

There are also kernel functions proposed for particular applications, such as natural language processing [24] and bioinformatics [31].

Selecting the kernel function $k(·,·)$ and its parameters (e.g., q or s) is an important issue in training. Generally, a cross-validation procedure is used to choose the best performing kernel function among a set of kernel functions on a separate validation set different from the training set. In recent years, multiple kernel learning (MKL) methods are proposed, where we use multiple kernels instead of selecting one specific kernel function and its corresponding parameters

$$k_{\eta }({\mathit{x}}_i,{\mathit{x}}_j)=f_{\eta }(\{k_m({\mathit{x}}_i^m,{\mathit{x}}_j^m){\}}_{m=1}^P)$$

where the combination function $f_{\eta }(·)$ can be a linear or a nonlinear function of the input kernels. Kernel functions, $\{k_m(·,·){\}}_{m=1}^P$, take P feature representations (not necessarily different) of data instances, where ${\mathit{x}}_i=\{{\mathit{x}}_i^m{\}}_{m=1}^P$, ${\mathit{x}}_i^m\in {\mathbb{R}}^{D_m}$, and D_m is the dimensionality of the corresponding feature representation.

The reasoning is similar to combining different classifiers: Instead of choosing a single kernel function and putting all our eggs in the same basket, it is better to have a set and let an algorithm do the picking or combination. There can be two uses of MKL: (i) Different kernels correspond to different notions of similarity and instead of trying to find which works best, a learning method does the picking for us, or may use a combination of them. Using a specific kernel may be a source of bias, and in allowing a learner to choose among a set of kernels, a better solution can be found. (ii) Different kernels may be using inputs coming from different representations possibly from different sources or modalities. Since these are different representations, they have different measures of similarity corresponding to different kernels. In such a case, combining kernels is one possible way to combine multiple information sources.

Since their original conception, there is significant work on the theory and application of multiple kernel learning. Fixed rules use the combination function in (1) as a fixed function of the kernels, without any training. Once we calculate the combined kernel, we train a single kernel machine using this kernel. For example, we can obtain a valid kernel by taking the summation or multiplication of two kernels as follows [10]:

$$k_{\eta }({\mathit{x}}_i,{\mathit{x}}_j)=k_1({\mathit{x}}_i^1,{\mathit{x}}_j^1)+k_2({\mathit{x}}_i^2,{\mathit{x}}_j^2)k_{\eta }({\mathit{x}}_i,{\mathit{x}}_j)=k_1({\mathit{x}}_i^1,{\mathit{x}}_j^1)k_2({\mathit{x}}_i^2,{\mathit{x}}_j^2).$$

The summation rule is applied successfully in computational biology [27] and optical digit recognition [25] to combine two or more kernels obtained from different representations.

Instead of using a fixed combination function, we can have a function parameterized by a set of parameters $\mathbf{\Theta }$ and then we have a learning procedure to optimize $\mathbf{\Theta }$ as well. The simplest case is to parameterize the sum rule as a weighted sum

$$k_{\eta }({\mathit{x}}_i,{\mathit{x}}_j|\mathbf{\Theta }=\mathit{\eta })=\sum _{m=1}^P{\eta }_m{k}_m({\mathit{x}}_i^m,{\mathit{x}}_j^m)$$

with ${\eta }_m\in \mathbb{R}$. Different versions of this approach differ in the way they put restrictions on the kernel weights [22], [4], [29], [19]. For example, we can use arbitrary weights (i.e., linear combination), nonnegative kernel weights (i.e., conic combination), or weights on a simplex (i.e., convex combination). A linear combination may be restrictive and nonlinear combinations are also possible [23], [13], [8]; our proposed approach is of this type and we will discuss these in more detail later.

We can learn the kernel combination weights using a quality measure that gives performance estimates for the kernel matrices calculated on training data. This corresponds to a function that assigns weights to kernel functions

$$\mathit{\eta }=g_{\eta }(\{k_m({\mathit{x}}_i^m,{\mathit{x}}_j^m){\}}_{m=1}^P).$$

The quality measure used for determining the kernel weights could be “kernel alignment” [21], [22] or another similarity measure such as the Kullback–Leibler divergence [36]. Another possibility inspired from ensemble and boosting methods is to iteratively update the combined kernel by adding a new kernel as training continues [5], [9]. In a trained combiner parameterized by $\mathbf{\Theta }$, if we assume $\mathbf{\Theta }$ to contain random variables with a prior, we can use a Bayesian approach. For the case of a weighted sum, we can, for example, have a prior on the kernel weights [11], [12], [28]. A recent survey of multiple kernel learning algorithms is given in [18].

This paper is organized as follows: We formulate our proposed nonlinear combination method localized MKL (LMKL) with detailed mathematical derivations in Section 2. We give our experimental results in Section 3 where we compare LMKL with MKL and single kernel SVM. In Section 4, we discuss the key properties of our proposed method together with related work in the literature. We conclude in Section 5.