A new scheme to learn a kernel in regularization networks

Abstract

In this paper, we propose a new scheme to learn a kernel function from the convex combination of finite given kernels in regularization networks. We show that the corresponding variational problem is convex and under certain conditions, the variational problem can be approximated by a semidefinite programming problem which coincides with the Micchelli and Pontil's (MP's) Model (Micchelli and Pontil, 2005 [10]).

Introduction

Kernel-based methods become very popular in the community of machine learning in recent years. Roughly speaking, a kernel-based method is constructing a nonlinear map from the data set to a Hilbert space (feature space), then building a linear algorithm in the feature space to implement the nonlinear counterpart in the data set. In many problems of machine learning, such as regression, classification and dimensionality reduction, kernel-based methods are very successful. However, the results depend on kernel severely, therefore how to select a ‘good’ kernel is a critical issue for all kernel-based methods. In this direction, there are some progresses. In general, the existing works can be divided into three classes based on tasks. Rayleigh coefficients play a key role in classification problems (see [7], [8], [12], [20]), as for regression, we recommend Micchelli and Pontil's works (see [2], [12]) and for dimensionality reduction, the celebrated work refers to paper [17]. In these works, convex optimization techniques, especially semidefinite programming, are basic tools (see [4], [15]). At the last of this paragraph, we want to mention that statistical generalization analysis of learning the kernel problem is also important. Recently, Ying and Campbell developed a novel generalization bound for learning the kernel problem, especially established satisfactory excess generalization bounds and misclassification error rates for learning Gaussian kernels (see [19]).

In this paper, we develop a scheme to learn an optimal kernel from the convex combination of finite given kernels in regularization networks. Before addressing our method, we review some basic notations.

For two given data sets $\mathcal{X}$ and $\mathcal{Y}$, the goal is learning a map from $\mathcal{X}$ to $\mathcal{Y}$ based on finite training data $\{(x_i,y_i){\}}_{i=1}^n\subset \mathcal{X}\times \mathcal{Y}$. In what follows, we will restrict $\mathcal{X}\subset {\mathbb{R}}^d$ and $\mathcal{Y}\subset \mathbb{R}$. A kernel K defined on $\mathcal{X}$ is a symmetric function from $\mathcal{X}\times \mathcal{X}$ to $\mathbb{R}$ satisfying that for any finite set $\{x_i{\}}_{i=1}^m$, the Gram matrix (kernel matrix) of order m

$$G_K=(K(x_i,x_j))$$

is positive semidefinite, and if the kernel matrix G_K is positive definite, we call K a positive definite kernel. What is more, for a given kernel K, there exists a unique reproducing kernel Hilbert space $H_K≔\overline{\mathrm{span}}\{K(x,·):x\in \mathcal{X}\}$ associated with K. The inner product of $H_K$, denoted as $〈·,·{〉}_K$, satisfies $〈f,K(x,·){〉}_K=f(x)$, for any $f\in H_K$, and we use $\parallel ·{\parallel }_K$ to represent the norm of H_K. For more details on kernels and reproducing kernel Hilbert spaces, see [1], [13].

Classical regularization network theory formulates the regression problem as a variational problem of finding a function f that minimizes the functional

$$\underset{f\in H_K}{\min }{Q}_K(f)≔\sum _{i=1}^n(f(x_i)-y_i{)}^2+\lambda \parallel f{\parallel }_K^2\text{,}$$

where $\lambda >0$ is the regularization parameter. It is well known that (see [5], [6], [13]) if f_K is a minimizer of (1.1), it has the form

$$f_K(x)=\sum _{i=1}^n{c}_{i}K(x_i,x),x\in \mathcal{X}\text{,}$$

for some real vector $C≔(c_1,c_2,\dots ,c_n{)}^T$ which is determined by $(\lambda I+G_K)C=Y$, $Y≔(y_1,y_2,\dots ,y_n{)}^T$. This classical regularization network theory is based on an essential assumption: the function f from $\mathcal{X}$ to $\mathcal{Y}$ lies in $H_K$ or can be well approximated by some element of H_K. See [11] for the approximation ability of H_K.

In this paper, a new scheme is proposed to learn an optimal kernel, that is,

$$\underset{K\in \mathcal{K}}{\min }\sum _{i=1}^n(f_K(x_i)-y_i{)}^2\text{,}$$

where $\mathcal{K}$ is the set of convex combination of finite given kernels. Our method is motivated by Micchelli and Pontil's work [10], in which the idea can be addressed as follows:

$$\underset{K\in \mathcal{K}}{\min }{Q}_K(f_K)\text{.}$$

The relation between these two models will be discussed in Section 3, as we will see, under certain conditions, the problem (1.3) can be approximated by a semidefinite programming problem which coincides with (1.4).

This paper is organized as follows: in Section 2, we address the basic issues of optimization problem (1.3): the existence of the solution and the convexity of this optimization problem. In Section 3, the relation between our model and MP's is discussed and we summarize our work in Section 4.