Fast Gaussian kernel support vector machine recursive feature elimination algorithm

Abstract

Gaussian kernel support vector machine recursive feature elimination (GKSVM-RFE) is a method for feature ranking in a nonlinear way. However, GKSVM-RFE suffers from the issue of high computational complexity, which hinders its applications. This paper investigates the issue of computational complexity in GKSVM-RFE, and proposes two fast versions for GKSVM-RFE, called fast GKSVM-RFE (FGKSVM-RFE), to speed up the procedure of recursive feature elimination in GKSVM-RFE. For this purpose, we design two kinds of ranking scores based on the first-order and second-order approximate schemes by introducing approximate Gaussian kernels. In iterations, FGKSVM-RFE fast calculates approximate ranking scores according to approximate schemes and ranks features based on approximate ranking scores. Experimental results reveal that our proposed methods can faster perform feature ranking than GKSVM-RFE and have compared performance to GKSVM-RFE.

Appendices

Appendix

Proof of Theorem 1

Proof

Let $\{\textbf {x}_{i},y_{i}\}_{i=1}^{n}$ be the training set, and S be the index set of candidate features. Given the solution α_i,i = 1,⋯ ,n to (4) and the Gaussian kernel parameter γ, we can calculate the first-order approximate ranking score $\tilde {c}^{(1)}_{m}$ for feature m using the first-order approximate Gaussian kernel (11). Namely,

$$ \begin{array}{@{}rcl@{}} \tilde{c}^{(1)}_{m} &=& \frac{1}{2}\left|\boldsymbol{\alpha}^{T}\tilde{\textbf{H}}_{1}^{S}\boldsymbol{\alpha}-\boldsymbol{\alpha}^{T}\tilde{\textbf{H}}_{1}^{S-\{m\}}\boldsymbol{\alpha}\right| \\ ~ &=& \frac{1}{2}\left|\sum\limits_{i=1}^{n}\sum\limits_{j=1}^{n} \left( \alpha_{i} \alpha_{j} y_{i} y_{j} \tilde{K}^{S}_{ij}-\alpha_{i} \alpha_{j} y_{i} y_{j} \tilde{K}_{ij}^{S-\{m\}}\right) \right| \end{array} $$

(21)

where $\tilde {K}^{S}_{ij}=\tilde {k}_{1}\left (\textbf {x}^{S}_{i},\textbf {x}^{S}_{j}\right )$ and $\tilde {K}_{ij}^{S-\{m\}}=\tilde {k}_{1}\left (\textbf {x}^{S-\{m\}}_{i},\textbf {x}^{S-\{m\}}_{j}\right )$.

According to the absolute value inequality theorem, we bound $\tilde {c}^{(1)}_{m}$ as follows:

$$ \begin{array}{@{}rcl@{}} \tilde{c}^{(1)}_{m} & \leq & \frac{1}{2}\sum\limits_{i=1}^{n}\sum\limits_{j=1}^{n} \left|\alpha_{i} \alpha_{j} y_{i} y_{j} \left( \tilde{K}^{S}_{ij}-\tilde{K}_{ij}^{S-\{m\}}\right) \right| \end{array} $$

(22)

$$ \begin{array}{@{}rcl@{}} &\leq& \frac{1}{2}\sum\limits_{i=1}^{n}\sum\limits_{j=1}^{n} \alpha_{i} \alpha_{j} \left| \tilde{K}^{S}_{ij}-\tilde{K}_{ij}^{S-\{m\}}\right| \end{array} $$

(23)

$$ \begin{array}{@{}rcl@{}} &\leq & \frac{1}{2}\sum\limits_{i=1}^{n}\sum\limits_{j=1}^{n} \alpha_{i} \alpha_{j} \left( \left| \tilde{K}^{S}_{ij}\right|+\left|\tilde{K}_{ij}^{S-\{m\}}\right|\right) \end{array} $$

(24)

Further, we can expand the approximate kernel function by

$$ \begin{array}{@{}rcl@{}} \left|\tilde{K}_{ij}^{S}\right|&=&\left|e^{-\gamma\left( \|\overline{\textbf{x}}_{i}\|_{2}^{2}+\|\overline{\textbf{x}}_{j}\|_{2}^{2}\right)} \left( 1+2\gamma\overline{\textbf{x}}_{i}^{T}\overline{\textbf{x}}_{j}\right)\right| \end{array} $$

(25)

$$ \begin{array}{@{}rcl@{}} &\leq& \left| e^{-\gamma\left( \|\overline{\textbf{x}}_{i}\|_{2}^{2}+\|\overline{\textbf{x}}_{j}\|_{2}^{2}\right)}\right| \left| 1+2\gamma\overline{\textbf{x}}_{i}^{T}\overline{\textbf{x}}_{j}\right| \end{array} $$

(26)

$$ \begin{array}{@{}rcl@{}} &\leq& \left|1+2\gamma\overline{\textbf{x}}_{i}^{T}\overline{\textbf{x}}_{j}\right| \end{array} $$

(27)

where $\overline {\mathbf {x}}_{i}$ denotes x_i with features in S, or $\overline {\mathbf {x}}_{i}=\mathbf {x}^{S}_{i}$.

Similarly, we can have the upper bound for $\left |\tilde {K}^{S-\{m\}}_{ij}\right |$. Namely,

$$ \begin{array}{@{}rcl@{}} \left|\tilde{K}^{S-\{m\}}_{ij}\right|\leq \left|1+2\gamma\left( \overline{\textbf{x}}^{(-m)}_{i}\right)^{T}\overline{\textbf{x}}^{(-m)}_{j}\right| \end{array} $$

(28)

where $\overline {\mathbf {x}}^{(-m)}_{i}$ denotes x_i with all features in (S −{m}),

We can rewrite $\left (\overline {\textbf {x}}^{(-m)}_{i}\right )^{T}\left (\overline {\textbf {x}}^{(-m)}_{j}\right )$ in (28) as follows.

$$ \left( \overline{\textbf{x}}^{(-m)}_{i}\right)^{T}\left( \overline{\textbf{x}}^{(-m)}_{j}\right)=\overline{\textbf{x}}_{i}^{T}\overline{\textbf{x}}_{j}-{x_{i}^{m}} {x_{j}^{m}} $$

(29)

Substituting (29) into (28), we get

$$ \begin{array}{@{}rcl@{}} \left|\tilde{K}^{S-\{m\}}_{ij}\right|&\leq& \left|1+2\gamma\overline{\textbf{x}}_{i}^{T}\overline{\textbf{x}}_{j}-2\gamma {x_{i}^{m}} {x_{j}^{m}} \right| \end{array} $$

(30)

$$ \begin{array}{@{}rcl@{}} &\leq& \left|1+2\gamma\overline{\textbf{x}}_{i}^{T}\overline{\textbf{x}}_{j}\right|+\left|2\gamma {x_{i}^{m}} {x_{j}^{m}} \right| \end{array} $$

(31)

Substituting (27) and (31) into (24), we can obtain

$$ \begin{array}{@{}rcl@{}} \tilde{c}^{(1)}_{m} &\leq & \frac{1}{2}\sum\limits_{i=1}^{n}\sum\limits_{j=1}^{n} \alpha_{i} \alpha_{j} \left( 2\left|1+2\gamma\overline{\textbf{x}}_{i}^{T}\overline{\textbf{x}}_{j}\right|+\left|2\gamma {x_{i}^{m}} {x_{j}^{m}} \right|\right) \end{array} $$

(32)

$$ \begin{array}{@{}rcl@{}} &=&\sum\limits_{i=1}^{n}\sum\limits_{j=1}^{n} \alpha_{i} \alpha_{j} \left( \left|1+2\gamma\overline{\textbf{x}}_{i}^{T}\overline{\textbf{x}}_{j}\right|+\left|\gamma {x_{i}^{m}} {x_{j}^{m}} \right|\right) \end{array} $$

(33)

which completes the proof of this theorem. □