New fast feature selection methods based on multiple support vector data description

Abstract

Feature selection can sort out useful features to obtain good performance when dealing with high-dimensional data. Feature selection methods based on support vector data description (SVDD) have been proposed for one-class classification problems: SVDD-radius-recursive feature elimination (SVDD-RRFE) and SVDD-dual-objective-recursive feature elimination (SVDD-DRFE). However, both SVDD-RRFE and SVDD-DRFE use only one-class samples even given a multi-class classification task, and suffer from high computational complexity. To remedy it, this paper extends both SVDD-RRFE and SVDD-DRFE to binary and multi-class classification problems using multiple SVDD models, and proposes fast feature ranking schemes for them in the case of the linear kernel. Experimental results on toy, UCI and microarray datasets show the efficiency and the feasibility of the proposed methods.

Appendix

1.1 A. Proof of Theorem 1

Proof

Assume that a linear SVDD model has been trained. Then we can get the center a of the hypersphere, and the set of support vectors SV . Substituting J _r (6) and ${J_{r}^{k}}$(7) into the radius ranking score J R _k ,we have

$$\begin{array}{@{}rcl@{}} JR_{k}&=& J_{r}-{J_{r}^{k}} \\ &=& \sum\limits_{{{\textbf{x}}_{sv}} \in {{SV}}} {\frac{{{R^{2}}\left( {{{\textbf{x}}_{sv}}} \right)}}{{\left| {SV} \right|}}}- \sum\limits_{{{\textbf{x}}_{sv}} \in {{SV}}} {\frac{{{R^{2}}\left( {{{\textbf{x}}^{k}_{sv}}} \right)}}{{\left| {SV} \right|}}} \end{array} $$

(31)

where ${\textbf {x}}_{sv}\in \mathbb {R}^{D}$is a supportvector, and ${\textbf {x}}^{k}_{sv}=[x_{(sv,1)},\cdots ,x_{(sv,k-1)},x_{(sv,k+1)},\cdots ,x_{(sv,D)}]^{T} \in \mathbb {R}^{D-1}$.

According to (31), it is necessary to find the difference $R^{2}({\textbf {x}}_{sv})-R^{2}({\textbf {x}}^{k}_{sv})$.Let $\textbf {a}^{k}=[a_{1},\cdots ,a_{k-1},a_{k+1},\cdots ,a_{D}] \in \mathbb {R}^{D-1}$.Then, we have

$$\begin{array}{@{}rcl@{}} &&R^{2}({\textbf{x}}_{sv})-R^{2}({\textbf{x}}^{k}_{sv})\\ &=&\|{\textbf{x}}_{sv}-\textbf{a}\|^{2}-\|{\textbf{x}}^{k}_{sv}-\textbf{a}^{k}\|^{2} \\ &=& {\textbf{x}}^{T}_{sv}{\textbf{x}}_{sv}-2{\textbf{x}}^{T}_{sv}{\textbf{a}} +\textbf{a}^{T}\textbf{a}-\\ &&\left( ({\textbf{x}}^{k}_{sv})^{T}{\textbf{x}}^{k}_{sv}-2({\textbf{x}}^{k}_{sv})^{T}{\textbf{a}^{k}} +(\textbf{a}^{k})^{T}\textbf{a}^{k} \right) \end{array} $$

(32)

Since

$$\begin{array}{@{}rcl@{}} {\textbf{x}}^{T}_{sv}{\textbf{x}}_{sv}-({\textbf{x}}^{k}_{sv})^{T}{\textbf{x}}^{k}_{sv}=x^{2}_{(sv,k)} \end{array} $$

(33)

$$\begin{array}{@{}rcl@{}} {\textbf{x}}^{T}_{sv}{\textbf{a}}-({\textbf{x}}^{k}_{sv})^{T}{\textbf{a}^{k}}=x_{(sv,k)}a_{k} \end{array} $$

(34)

and

$$\begin{array}{@{}rcl@{}} \textbf{a}^{T}\textbf{a}-(\textbf{a}^{k})^{T}\textbf{a}^{k}={a^{2}_{k}} \end{array} $$

(35)

we substitute (33), (34) and (35) into (32), and get

$$\begin{array}{@{}rcl@{}} R^{2}({\textbf{x}}_{sv})-R^{2}({\textbf{x}}^{k}_{sv})=x^{2}_{(sv,k)}-2x_{(sv,k)}a_{k}+{a^{2}_{k}} \end{array} $$

(36)

Then, substituting (36) into (31), the radius ranking score can be rewritten as:

$$\begin{array}{@{}rcl@{}} JR_{k}=\frac{1}{|SV|}\sum\limits_{{\textbf{x}}_{sv}\in SV} \left( x^{2}_{(sv,k)}-2x_{(sv,k)}a_{k}+{a^{2}_{k}}\right) \end{array} $$

(37)

This completes the proof of Theorem 1. □

1.2 B. Proof of Theorem 2

Proof

Assume that a linear SVDD model has been trained. Then we can get the coefficients α _i , i = 1,⋯ ,n of the hypersphere, and the set of support vectors S V = {α _i |α _i > 0,i = 1,⋯ ,n}.Substituting J _d (9) and ${J_{d}^{k}}$(10) into the dual-objective ranking score J D _k ,we have

$$\begin{array}{@{}rcl@{}} JD_{k}&=& J_{d}-{J_{d}^{k}} \\ &=& \sum\limits_{i=1}^{n}\alpha_{i}{\textbf{x}}_{i}^{T}{\textbf{x}}_{i} -\sum\limits_{i=1}^{n}\sum\limits_{j=1}^{n}\alpha_{i}\alpha_{j}{\textbf{x}}_{i}^{T}{\textbf{x}}_{j}- \\ &&\sum\limits_{i=1}^{n}\alpha_{i}({\textbf{x}}^{k}_{i})^{T}{\textbf{x}}^{k}_{i} -\sum\limits_{i=1}^{n}\sum\limits_{j=1}^{n}\alpha_{i}\alpha_{j}({\textbf{x}}^{k}_{i})^{T}{\textbf{x}}^{k}_{j} \end{array} $$

(38)

Since

$$\begin{array}{@{}rcl@{}} {\textbf{x}}_{i}^{T}{\textbf{x}}_{i}-({\textbf{x}}^{k}_{i})^{T}{\textbf{x}}^{k}_{i}=x^{2}_{(i,k)} \end{array} $$

(39)

and

$$\begin{array}{@{}rcl@{}} {\textbf{x}}_{i}^{T}{\textbf{x}}_{j}-({\textbf{x}}^{k}_{i})^{T}{\textbf{x}}^{k}_{j}=x_{(i,k)}x_{(j,k)} \end{array} $$

(40)

we substitute (39) and (40) into (38), and get

$$\begin{array}{@{}rcl@{}} JD_{k}=\sum\limits_{i=1}^{n}\alpha_{i} x^{2}_{(i,k)} -\sum\limits_{i=1}^{n}\sum\limits_{j=1}^{n}\alpha_{i}\alpha_{j} x_{(i,k)}x_{(j,k)} \end{array} $$

(41)

Since only support vectors contribute to computing (41), we rewrite (41) as follows:

$$\begin{array}{@{}rcl@{}} JD_{k}&=&\sum\limits_{{\textbf{x}}_{sv}\in SV} \alpha_{sv} x^{2}_{(sv,k)}- \\ &&\sum\limits_{{\textbf{x}}_{sv}\in SV} \sum\limits_{{\textbf{x}}_{sv^{\prime}}\in SV}\alpha_{sv}\alpha_{sv^{\prime}} x_{(sv,k)}x_{(sv^{\prime},k)} \end{array} $$

(42)

This completes the proof of Theorem 2. □