Generalization-error-bound-based discriminative dictionary learning

Abstract

Support vector guided dictionary learning, as a discriminative dictionary learning method combining with support vector machine (SVM), embodies the margin maximization principle and achieves good generalization performances in many practical applications. However, this method ignores the key fact that the generalization performance of the SVM classifier depends not only on the margin between two classes of training samples, but also on the radius of the smallest sphere covering them. In the paper, we propose a novel method called generalization-error-bound-based discriminative dictionary learning (GEBDDL). The basic insight of GEBDDL is that the coding vectors, which are used to build the SVM classifier, are not fixed during the learning process. As a result, the radius of the smallest sphere changes with the learned coding vectors. The key feature of GEBDDL is that it explicitly incorporates the radius-margin-bound, which is directly related to the upper bound of the leave-one-out error of SVM, into its objective function to guide learning the dictionary and the coding vectors, and building the SVM classifier. In the paper, we first elaborate our motivation and propose the optimization model and then discuss how to solve it in detail. Further, we explore how to approximate the radius of the smallest sphere in our methodology. This can enhance the computational efficiency by bypassing the quadratic programming problem of computing the radius, while yielding a close performance to GEBDDL. Finally, the comprehensive experiments are conducted on several benchmark datasets, and the results demonstrate the superiority of the proposed methods over the other competing methods.

Appendix

Given formula (25)

$$\begin{aligned}&{{{{\mathcal {F}}}}_1}(\mathbf{{S}}) = \left\| {\mathbf{{X}} - \mathbf{{DS}}} \right\| _2^2 + {\lambda _1}\left\| \mathbf{{S}} \right\| _2^2 \nonumber \\&{} { {{{{\mathcal {F}}}}_2}(\mathbf{{S}}) = \hat{R}^{2} {} {}} \nonumber \\&{}=\max _{\beta _{i}} \sum _{i=1}^{N} \beta _{i} \tilde{k}\left( {\mathbf {s}}_{i}, {\mathbf {s}}_{i}\right) -\sum _{i=1}^{N} \sum _{j=1}^{N} \beta _{i} \beta _{j} \tilde{k}\left( {\mathbf {s}}_{i}, {\mathbf {s}}_{i}\right) \nonumber \\&{}={\text {tr}}\left( {\mathbf {S}} {\varvec{\Lambda }} {\mathbf {S}}^{T}+\frac{1}{2 \theta } {\mathbf {e}} \varvec{\beta }\right) -\varvec{\beta }^{T}\left( {\mathbf {S}}^{T} {\mathbf {S}}+\frac{1}{2 \theta } {\mathbf {I}}\right) \varvec{\beta } \nonumber \\&{} {{{{\mathcal {F}}}}_3}(\mathbf{{S}}) = \sum \limits _{c = 1}^C {\left( {{{\left\| {{\mathbf{{w}}_c}} \right\| }^2}\mathrm{{ + }}\theta \sum \limits _{i = 1}^N {{\delta _i}{{\left( {1 - y_i^c(\mathbf{{w}}_c^T{\mathbf{{s}}_i} + {b_c})} \right) }^2}} } \right) } \nonumber \\ \end{aligned}$$

(47)

The detailed derivation is as follows, first of all, for ${\mathcal{F}_1}(\mathbf{{S}})$, we have

$$\begin{aligned} \begin{aligned} {\mathcal {F}}_{1}({\mathbf {S}})&=({\mathbf {X}}-{\mathbf {D}} {\mathbf {S}})^{T}({\mathbf {X}}-\mathbf {D S}) +\lambda _{1} {\mathbf {S}}^{T} {\mathbf {S}}\\&=\left( {\mathbf {X}}^{T} {\mathbf {X}}-{\mathbf {X}}^{T} {\mathbf {D}} {\mathbf {S}}-{\mathbf {S}}^{T} {\mathbf {D}}^{T} {\mathbf {D}} {\mathbf {S}}\right) +\lambda _{1}{\mathbf {S}}^{T} {\mathbf {S}} \end{aligned} \end{aligned}$$

(48)

The partial derivatives of ${\mathcal {F}}_{1}({\mathbf {S}})$ with respect to ${\mathbf {S}}$ can be formulated as

$$\begin{aligned} \frac{{\partial {{{{\mathcal {F}}}}_1}(\mathbf{{S}})}}{{\partial \mathbf{{S}}}} = 2{\mathbf{{D}}^T}{} \mathbf{{DS}} - 2{\mathbf{{D}}^T}{} \mathbf{{X}} + 2{\lambda _1}{} \mathbf{{S}} \end{aligned}$$

(49)

The partial derivatives of ${\mathcal {F}}_{2}({\mathbf {S}})$ with respect to ${\mathbf {S}}$ can be formulated as

$$\begin{aligned} \begin{aligned} \frac{\partial {\mathcal {F}}_{2}({\mathbf {S}})}{\partial {\mathbf {S}}}&=\frac{\partial \left( {\text {tr}}\left( {\mathbf {S}} {\varvec{\Lambda }} {\mathbf {S}}^{T}\right) -\varvec{\beta }^{T} {\mathbf {S}}^{T} {\mathbf {S}} \varvec{\beta }\right) }{\partial {\mathbf {S}}}\\&=\frac{\partial \left( {\text {tr}}\left( {\mathbf {S}} {\varvec{\Lambda }} {\mathbf {S}}^{T}\right) -{\text {tr}}\left( \varvec{\beta }^{T} {\mathbf {S}}^{T} {\mathbf {S}} \varvec{\beta }\right) \right) }{2 {\mathbf {S}}}\\&=2\mathbf{S}{\varvec{\Lambda }} - 2\varvec{\beta }{\varvec{\beta }}^T\mathbf{{S}} \end{aligned} \end{aligned}$$

(50)

For ${{{{\mathcal {F}}}}_3}(\mathbf{{S}})$, there are two cases for discussion. First, when $y_i^c(\mathbf{{w}}_c^T{\mathbf{{s}}_i} + {b_c})<1$,

$$\begin{aligned} {{{{\mathcal {F}}}}_3}(\mathbf{{S}}) = \sum _{c=1}^{C} \min _{{\mathbf {w}}_{c}, b_{c}}\left( \begin{array}{l} \left\| {\mathbf {w}}_{c}\right\| ^{2}+n\left( 1+b_{c}^{2}\right) +{\mathbf {w}}_{c}^{T}\left( \mathbf {S S}^{T}\right) {\mathbf {w}}_{c} \\ +2 {\mathbf {w}}_{c}(\mathbf {S e})^{T} b_{c}-2 {\mathbf {w}}_{c}^{T} {\mathbf {S}} {\mathbf {y}}_{c}-2 {\mathbf {y}}_{c}^{T} {\mathbf {e}} b_{c} \end{array}\right) \end{aligned}$$

(51)

When $y_{i}\left( {\mathbf {w}}_{c}^{T} {\mathbf {s}}_{i}+b_{c}\right) \,\ge \,q 1$

$$\begin{aligned} {\mathcal {F}}_{3}({\mathbf {S}})=\sum _{c=1}^{C} \min _{{\mathbf {w}}_{c}, b_{c}}\left\| {\mathbf {w}}_{c}\right\| ^{2} \end{aligned}$$

(52)

Hence, the partial derivatives of ${\mathcal {F}}_{3}({\mathbf {S}})$ with respect to ${\mathbf {S}}$ can be formulated as

$$\begin{aligned} \frac{\partial {\mathcal {F}}_{3}({\mathbf {S}})}{\partial {\mathbf {S}}}=\sum _{c=1}^{C}\left( 2 {\mathbf {w}}_{c} {\mathbf {w}}_{c}^{T} {\mathbf {S}}+b_{c} {\mathbf {w}}_{c} {\mathbf {e}}^{T}-{\mathbf {w}}_{c} {\mathbf {y}}_{c}^{T}\right) \end{aligned}$$

(53)