Privacy preserving and fast decision for novelty detection using support vector data description

Abstract

Support vector data description (SVDD) has been widely used in novelty detection applications. Since the decision function of SVDD is expressed through the support vectors which contain sensitive information, the support vectors will be disclosed when SVDD is used to detect the unknown samples. Accordingly, privacy concerns arise. In addition, when it is applied to large datasets, SVDD does not scale well as its complexity is linear with the size of the training dataset (actually the number of support vectors). Our work here is distinguished in two aspects. First, by decomposing the kernel mapping space into three subspaces and exploring the pre-image of the center of SVDD’s sphere in the original space, a fast decision approach of SVDD, called FDA-SVDD, is derived, which includes three implementation versions, called FDA-SVDD-I, FDA-SVDD-II and FDA-SVDD-III. The decision complexity of the proposed method is reduced to only $O$(1). Second, as the decision function of FDA-SVDD only refers to the pre-image of the sphere center, the privacy of support vectors can be preserved. Therefore, the proposed FDA-SVDD is particularly attractive in privacy-preserving novelty detection applications. Empirical analysis conducted on UCI and USPS datasets demonstrates the effectiveness of the proposed approach and verifies the derived theoretical results.

Appendix

Proof of Theorem 4:

According to Eq. (16) and Eq. (19), we have

$$\begin{aligned} \text {ISE}({\varvec{w}})&\approx \mathop {\sum }\limits _{{\varvec{x}}_i \in \chi _\mathrm{in} \cup \chi _\mathrm{on}} \mathop {\sum }\limits _{{{\varvec{x}}_i \in \chi _\mathrm{in} \cup \chi _\mathrm{on} }} w_i w_j k({\varvec{x}}_i ,{\varvec{x}}_j )\nonumber \\&\quad -2\mathop {\sum }\limits _{{\varvec{x}}_i \in \chi _{in} \cup \chi _{on}} {w_i \sum \limits _{{\varvec{x}}_j \in SV} {\alpha _j k({\varvec{x}}_i ,{\varvec{x}}_j )} }\nonumber \\&\quad +\sum \limits _{{\varvec{x}}_i \in SV} {\sum \limits _{{\varvec{x}}_j \in SV} {\alpha _i \alpha _j k({\varvec{x}}_i ,{\varvec{x}}_j )} } \end{aligned}$$

(23)

$$\begin{aligned} w_i =\frac{\sum \nolimits _{{\varvec{x}}_j \in SV} {\alpha _j k({\varvec{x}}_i ,{\varvec{x}}_j )} }{\sum \nolimits _{{\varvec{x}}_j \in \chi _{in} \cup \chi _{on} } {k({\varvec{x}}_i ,{\varvec{x}}_j )} }. \end{aligned}$$

(24)

Then,

$$\begin{aligned}&\frac{\partial ISE({\varvec{w}})}{\partial w_k }=2\varphi ({\varvec{x}}_k )^T\sum \limits _{{\varvec{x}}_i \in \chi _{in} \cup \chi _{on} } {w_i \varphi ({\varvec{x}}_i )} -2\varphi ({\varvec{x}}_k )^T\nonumber \\&\quad \times \sum \limits _{{\varvec{x}}_j \in SV} {\alpha _j \varphi ({\varvec{x}}_j )}\nonumber \\&=2\varphi ({\varvec{x}}_k )^T\left( {\sum \limits _{{\varvec{x}}_i \in \chi _{in} \cup \chi _{on} } {w_i \varphi ({\varvec{x}}_i )} -\sum \limits _{{\varvec{x}}_j \in SV} {\alpha _j \varphi ({\varvec{x}}_j )} }\right) . \nonumber \\ \end{aligned}$$

(25)

Substitute Eq. (24) to Eq. (25), i.e.,

$$\begin{aligned}&\frac{\partial ISE({\varvec{w}})}{\partial w_k }\nonumber \\&\quad =2\varphi ({\varvec{x}}_k )^T\left( {\sum \limits _{{\varvec{ x}}_i \in \chi _{in} \cup \chi _{on} } {\frac{\sum \nolimits _{{\varvec{ x}}_j \in SV} {\alpha _j k({\varvec{ x}}_i ,{\varvec{ x}}_j )} }{\sum \nolimits _{{\varvec{ x}}_j \in \chi _{in} \cup \chi _{on} } {k({\varvec{ x}}_i ,{\varvec{x}}_j )} }\varphi ({\varvec{ x}}_i )} -\sum \limits _{{\varvec{x}}_j \in SV} {\alpha _j \varphi ({\varvec{x}}_j )} }\right) \nonumber \\&\quad =2\varphi ({\varvec{ x}}_k )^T\left( {\sum \limits _{{\varvec{ x}}_i \in \chi _{in} \cup \chi _{on} } {\frac{\sum \nolimits _{{\varvec{x}}_j \in SV} {\alpha _j \varphi ({\varvec{ x}}_i )\varphi ({\varvec{ x}}_j )} }{\sum \nolimits _{{\varvec{ x}}_j \in \chi _{in} \cup \chi _{on} } {\varphi ({\varvec{x}}_i )\varphi ({\varvec{ x}}_j )} }\varphi ({\varvec{ x}}_i )} -\sum \limits _{{\varvec{ x}}_j \in SV} {\alpha _j \varphi ({\varvec{ x}}_j )} }\right) \nonumber \\&\quad =2\varphi ({\varvec{x}}_k )^T\left( {\sum \limits _{{\varvec{ x}}_i \in \chi _{in} \cup \chi _{on} } {\frac{\sum \nolimits _{{\varvec{ x}}_j \in SV} {\alpha _j \varphi ({\varvec{ x}}_i )\varphi ({\varvec{ x}}_j )} }{\sum \nolimits _{{\varvec{x}}_j \in \chi _{in} \cup \chi _{on} } {\varphi ({\varvec{ x}}_j )} }} -\sum \limits _{{\varvec{ x}}_j \in SV} {\alpha _j \varphi ({\varvec{x}}_j )} }\right) \nonumber \\&\quad =2\varphi ({\varvec{x}}_k )^T\left( {\frac{\sum \nolimits _{{\varvec{ x}}_i \in \chi _{in} \cup \chi _{on} } {\sum \nolimits _{{\varvec{ x}}_j \in SV} {\alpha _j \varphi ({\varvec{x}}_i )\varphi ({\varvec{ x}}_j )} } }{\sum \nolimits _{{\varvec{ x}}_j \in \chi _{in} \cup \chi _{on} } {\varphi ({\varvec{ x}}_j )} }-\sum \limits _{{\varvec{ x}}_j \in SV} {\alpha _j \varphi ({\varvec{x}}_j )} }\right) \nonumber \\&\quad =2\varphi ({\varvec{ x}}_k )^T\left( {\frac{\sum \nolimits _{{\varvec{x}}_i \in \chi _{in} \cup \chi _{on} } {\varphi ({\varvec{ x}}_i )} }{\sum \nolimits _{{\varvec{ x}}_j \in \chi _{in} \cup \chi _{on} } {\varphi ({\varvec{x}}_j )} }\sum \limits _{{\varvec{x}}_j \in SV} {\alpha _j \varphi ({\varvec{ x}}_j )} -\sum \limits _{{\varvec{ x}}_j \in SV} {\alpha _j \varphi ({\varvec{ x}}_j )} }\right) \nonumber \\&\quad =2\left( {\sum \limits _{{\varvec{ x}}_j \in SV} {\alpha _j k({\varvec{x}}_j ,{\varvec{ x}}_k )} -\sum \limits _{{\varvec{ x}}_j \in SV} {\alpha _j k({\varvec{x}}_j ,\mathrm{\mathbf{x}}_k )} }\right) \nonumber \\&\quad =0.\end{aligned}$$

(26)

Clearly, Theorem 4 holds.