A Novel Multiple Kernel Learning Method Based on the Kullback–Leibler Divergence

Abstract

In this paper we develop a novel multiple kernel learning (MKL) model that is based on the idea of the multiplicative perturbation of data in a new feature space in the framework of uncertain convex programs (UCPs). In the proposed model, we utilize the Kullback–Leibler divergence to measure the difference between the estimated kernel weights and ideal kernel weights. Instead of directly handling the proposed model in the primal, we obtain the optimistic counterpart of its Langrage dual in terms of the theory of UCPs and solve it by using the alternating optimization. In the case of a varying parameter, the proposed model gives the solution path from a robust combined kernel to some combined kernel corresponding to the initially ideal kernel weights. In addition, we also give a simple strategy to select the initial kernel weights as the ideal kernel weights if any prior knowledge of kernel weights is not available. Experimental results on several data sets show that the proposed model can obtain competitive performance with some of the state-of-the-art MKL algorithms.

Appendix: The derivation of Eq. (13)

$$\begin{aligned}&\mathop {\max }\limits _{{\varvec{\alpha }},{\varvec{\theta }}} f({\varvec{\alpha }},{\varvec{\theta }}):=-\frac{1}{2}\sum _{k=1}^d {\theta _k } {\varvec{\alpha }}^{T}diag({{\varvec{y}}})K_k diag({{\varvec{y}}}){\varvec{\alpha }}+{\varvec{\alpha }}^{T}{{\varvec{e}}}- \gamma \sum _{k=1}^d {\theta _k } \log \frac{\theta _k }{q_k }\nonumber \\&\hbox {s.t.} \sum _{i=1}^n {\alpha _i y_i =0} ,C\ge \alpha _i \ge 0,i=1,\ldots ,n,\nonumber \\&\quad \sum _{k=1}^d {\theta _k } =1,\theta _k \ge 0, \end{aligned}$$

(18)

From Eq. (18), for fixed $\alpha _i (i=1,\ldots ,n)$, $f({\varvec{\alpha }},{\varvec{\theta }})$ is strict concave with respect to $\theta$. Thus maximizing $f({\varvec{\alpha }},{\varvec{\theta }})$ over the simplex yields a unique solution. We define the following partial Lagrangian function.

$$\begin{aligned} L({\varvec{\theta }}):=-\frac{1}{2}\sum _{k=1}^d {\theta _k } {\varvec{\alpha }}^{T}diag({{\varvec{y}}})K_k diag({{\varvec{y}}}){\varvec{\alpha }}+{\varvec{\alpha }}^{T}{{\varvec{e}}}- \gamma \sum _{k=1}^d {\theta _k } \log \frac{\theta _k }{q_k }+\lambda \left( 1-\sum _{k=1}^d {\theta _k } \right) \nonumber \\ \end{aligned}$$

(19)

Setting the derivative of $L({\varvec{\theta }})$ with respect to $\theta _k$ to zero gives

$$\begin{aligned} \frac{\partial L({\varvec{\theta }})}{\partial \theta _k }= -\frac{1}{2}{\varvec{\alpha }}^{T}diag({{\varvec{y}}})K_k diag({{\varvec{y}}}){\varvec{\alpha }}-\gamma \log \theta _k +\gamma \log q_k -\lambda =0. \end{aligned}$$

(20)

From (20), one has

$$\begin{aligned} -\frac{1}{2}{\varvec{\alpha }}^{T}diag({{\varvec{y}}})K_k diag({{\varvec{y}}}){\varvec{\alpha }}-\lambda =\gamma \log \frac{\theta _k }{q_k } \end{aligned}$$

(21)

From Eq. (21), one has

$$\begin{aligned} q_k e^{-\frac{1}{2\gamma }{\varvec{\alpha }}^{T}diag({{\varvec{y}}})K_k diag({{\varvec{y}}}){\varvec{\alpha }}-\frac{\lambda }{\gamma }}=\theta _k . \end{aligned}$$

(22)

Note that $\sum _{k=1}^d {\theta _k } =1$. We have

$$\begin{aligned}&\sum _{k=1}^d {q_k } e^{-\frac{1}{2\gamma }{\varvec{\alpha }}^{T}diag({{\varvec{y}}})K_k diag({{\varvec{y}}}){\varvec{\alpha }}-\frac{\lambda }{\gamma }}=\sum _{k=1}^d {\theta _k } =1.\end{aligned}$$

(23)

$$\begin{aligned}&e^{-\frac{\lambda }{\gamma }}\sum _{k=1}^d {q_k } e^{-\frac{1}{2\gamma }{\varvec{\alpha }}^{T}diag({{\varvec{y}}})K_k diag({{\varvec{y}}}){\varvec{\alpha }}}=1. \end{aligned}$$

(24)

Substituting Eq. (24) into Eq. (22), one has

$$\begin{aligned} \theta _k =\frac{q_k e^{\frac{-\sum _{i,j} {\alpha _i \alpha _j y_i y_j K_k ({{\varvec{x}}}_{{\varvec{i}}} ,\mathbf{x}_\mathbf{j} )} }{2\gamma }}}{\sum _{s=1}^d {q_s e^{\frac{-\sum _{i,j} {\alpha _i \alpha _j y_i y_j K_s (\mathbf{x}_i ,\mathbf{x}_j )} }{2\gamma }}} },(k=1,\ldots ,d). \end{aligned}$$

(25)

From Eq. (25), one can see that the non-negativity of $\theta$ can be automatically guaranteed.