Iteratively local fisher score for feature selection

Abstract

In machine learning, feature selection is a kind of important dimension reduction techniques, which aims to choose features with the best discriminant ability to avoid the issue of curse of dimensionality for subsequent processing. As a supervised feature selection method, Fisher score (FS) provides a feature evaluation criterion and has been widely used. However, FS ignores the association between features by assessing all features independently and loses the local information for fully connecting within-class samples. In order to solve these issues, this paper proposes a novel feature evaluation criterion based on FS, named iteratively local Fisher score (ILFS). Compared with FS, the new criterion pays more attention to the local structure of data by using K nearest neighbours instead of all samples when calculating the scatters of within-class and between-class. In order to consider the relationship between features, we calculate local Fisher scores of feature subsets instead of scores of single features, and iteratively select the current optimal feature to achieve this idea like sequential forward selection (SFS). Experimental results on UCI and TEP data sets show that the improved algorithm performs well in classification activities compared with some other state-of-the-art methods.

Appendix A: Proof of Theorem 1

Proof

We adopt the mathematical induction method to prove Theorem 1. Fist, we generate the series {J(G(t))}, t = 1,⋯ ,m using the rule (5).

For t = 1, G(0) = ∅ and $\overline {G(0)}=\{1,\cdots ,m\}$. The highest score can be obtained by

$$ \begin{array}{@{}rcl@{}} J(G(1))=\frac{s_{b}^{G(1)}}{s_{w}^{G(1)}}=\max_{k\in\overline{G(0)}}\frac{s_{b}^{G(0)}+s^{\{k\}}_{b}}{s_{w}^{G(0)}+s^{\{k\}}_{w}}=\max_{k\in\overline{G(0)}}\frac{s^{\{k\}}_{b}}{s^{\{k\}}_{w}} \end{array} $$

(12)

where $s_{b}^{G(0)}=0$ and $s_{w}^{G(0)}=\delta$. In other words, J(G(1)) is generated by maximizing the local Fisher score of single feature. Then $G(1)=\{k_{1}^{*}\}$ according to (5), and $\overline {G(1)}=\{1,\cdots ,m\}-G(1)$, where $k_{1}^{*}$ is the optimal feature selected in the first iteration. From now, $k_{t}^{*}$ denotes the optimal feature selected in the t-th iteration.

For t = 2, J(G(2)) can be represented by

$$ J(G(2))=\frac{s_{b}^{G(2)}}{s_{w}^{G(2)}}=\frac{s_{b}^{G(1)}+s^{\{k_{2}^{*}\}}_{b}}{s_{w}^{G(1)}+s^{\{k_{2}^{*}\}}_{w}}=\max_{k\in\overline{G(1)}}\frac{s_{b}^{G(1)}+s^{\{k\}}_{b}}{s_{w}^{G(1)}+s^{\{k\}}_{w}} $$

(13)

Naturally, the score of the feature subset $\{k_{2}^{*}\}$ is less than or equal to that of the feature subset $\{k_{1}^{*}\}$. Namely

$$ \begin{array}{@{}rcl@{}} \frac{s^{\{k_{2}^{*}\}}_{b}}{s^{\{k_{2}^{*}\}}_{w}}\leq \frac{s_{b}^{G(1)}}{s_{w}^{G(1)}} \end{array} $$

(14)

By comparing (12) and (13), we have

$$ \begin{array}{@{}rcl@{}} J(G(2))-J(G(1))&=&\frac{s_{b}^{G(2)}}{s_{w}^{G(2)}}-\frac{s_{b}^{G(1)}}{s_{w}^{G(1)}}\\ &=&\frac{s_{b}^{G(1)}+s^{\{k_{2}^{*}\}}_{b}}{s_{w}^{G(1)}+s^{\{k_{2}^{*}\}}_{w}}-\frac{s_{b}^{G(1)}}{s_{w}^{G(1)}}\\ &=&\frac{s^{\{k_{2}^{*}\}}_{b}s_{w}^{G(1)}-s_{b}^{G(1)}s^{\{k_{2}^{*}\}}_{w}}{\left( s_{w}^{G(1)}+s^{\{k_{2}^{*}\}}_{w}\right)s_{w}^{G(1)}} \end{array} $$

(15)

Note that $s_{w}^{G(1)}>0$ and $s^{\{k_{2}^{*}\}}_{w}\geq 0$. Substituting (14) into (15), we have

$$ \begin{array}{@{}rcl@{}} J(G(2))-J(G(1))\leq 0 \end{array} $$

(16)

which means that Theorem 1 holds in the first two iterations.

Now, we assume that Theorem 1 holds in the (t − 1)-th and t-th iterations. Namely,

$$ \begin{array}{@{}rcl@{}} J(G(t))-J(G(t-1))\leq 0 \end{array} $$

(17)

For ∀t, J(G(t)) has the form:

$$ \begin{array}{@{}rcl@{}} J(G(t))&=&\frac{s_{b}^{G(t)}}{s_{w}^{G(t)}}=\frac{s_{b}^{G(t-1)}+s^{\{k_{t}^{*}\}}_{b}}{s_{w}^{G(t-1)}+s^{\{k_{t}^{*}\}}_{w}}\\&=&\max_{k\in\overline{G(t-1)}}\frac{s_{b}^{G(t-1)}+s^{\{k\}}_{b}}{s_{w}^{G(t-1)}+s^{\{k\}}_{w}} \end{array} $$

(18)

The difference of J(G(t)) and J(G(t + 1)) can be described as

$$ \begin{array}{@{}rcl@{}} J(G(t+1)) - J(G(t))&=&\frac{s_{b}^{G(t+1)}}{s_{w}^{G(t+1)}}-\frac{s_{b}^{G(t)}}{s_{w}^{G(t)}}\\ &=&\frac{s_{b}^{G(t)}+s^{\{k_{t+1}^{*}\}}_{b}}{s_{w}^{G(t)}+s^{\{k_{t+1}^{*}\}}_{w}}-\frac{s_{b}^{G(t)}}{s_{w}^{G(t)}}\\ &=&\frac{s^{\{k_{t+1}^{*}\}}_{b}s_{w}^{G(t)}-s_{b}^{G(t)}s^{\{k_{t+1}^{*}\}}_{w}}{\left( s_{w}^{G(t)}+s^{\{k_{t+1}^{*}\}}_{w}\right)s_{w}^{G(t)}} \end{array} $$

(19)

Because the denominator of (19) is greater than 0, we just consider the numerator of (19) that can be further represented as

$$ \begin{array}{@{}rcl@{}} &&s^{\{k_{t+1}^{*}\}}_{b}s_{w}^{G(t)}-s_{b}^{G(t)}s^{\{k_{t+1}^{*}\}}_{w}\\ &=& s^{\{k_{t+1}^{*}\}}_{b}\left( s_{w}^{G(t-1)}+s^{\{k_{t}^{*}\}}_{w}\right) -\left( s_{b}^{G(t-1)}+s^{\{k_{t}^{*}\}}_{b}\right)s^{\{k_{t+1}^{*}\}}_{w}\\ &=&\left( s^{\{k_{t+1}^{*}\}}_{b} s_{w}^{G(t-1)}-s_{b}^{G(t-1)} s^{\{k_{t+1}^{*}\}}_{w}\right)\\&&+\left( s^{\{k_{t+1}^{*}\}}_{b} s^{\{k_{t}^{*}\}}_{w}-s^{\{k_{t}^{*}\}}_{b} s^{\{k_{t+1}^{*}\}}_{w}\right) \end{array} $$

(20)

Because (17) holds true, we have the following inequalities

$$ \begin{array}{@{}rcl@{}} \frac{s_{b}^{G(t-1)}+s^{\{k\}}_{b}}{s_{w}^{G(t-1)}+s^{\{k\}}_{w}} \leq \frac{s_{b}^{G(t-1)}+s^{\{k_{t}^{*}\}}_{b}}{s_{w}^{G(t-1)}+s^{\{k_{t}^{*}\}}_{w}} \leq \frac{s_{b}^{G(t-1)}}{s_{w}^{G(t-1)}} \end{array} $$

(21)

where $k,k^{*}_{t}\in \overline {G(t-1)}$ and $k\neq k^{*}_{t}$. We separately derive the three inequalities and get

$$ \frac{s_{b}^{G(t-1)} + s^{\{k_{t}^{*}\}}_{b}}{s_{w}^{G(t-1)} + s^{\{k_{t}^{*}\}}_{w}} \!\leq\! \frac{s_{b}^{G(t-1)}}{s_{w}^{G(t-1)}} \!\Rightarrow\! s_{w}^{G(t-1)}s^{\{k_{t}^{*}\}}_{b}-s_{b}^{G(t-1)}s^{\{k_{t}^{*}\}}_{w} \!\leq\! 0 $$

(22)

$$ \frac{s_{b}^{G(t-1)}+s^{\{k\}}_{b}}{s_{w}^{G(t-1)}+s^{\{k\}}_{w}} \leq \frac{s_{b}^{G(t-1)}}{s_{w}^{G(t-1)}}\Rightarrow s_{w}^{G(t-1)}s^{\{k\}}_{b}-s_{b}^{G(t-1)}s^{\{k\}}_{w} \leq 0 $$

(23)

and

$$ \begin{array}{@{}rcl@{}} &&~~~\frac{s_{b}^{G(t-1)}+s^{\{k\}}_{b}}{s_{w}^{G(t-1)}+s^{\{k\}}_{w}} \leq \frac{s_{b}^{G(t-1)}+s^{\{k_{t}^{*}\}}_{b}}{s_{w}^{G(t-1)}+s^{\{k_{t}^{*}\}}_{w}} \\ &&\Rightarrow \left( s^{\{k\}}_{b} s^{\{k_{t}^{*}\}}_{w}-s^{\{k_{t}^{*}\}}_{b} s^{\{k\}}_{w} \right)+\left( s^{\{k\}}_{b} s_{w}^{G(t-1)}-s_{b}^{G(t-1)} s^{\{k\}}_{w}\right) \\ &&\leq \left( s_{w}^{G(t-1)}s^{\{k_{t}^{*}\}}_{b}-s_{b}^{G(t-1)}s^{\{k_{t}^{*}\}}_{w}\right) \leq 0 \end{array} $$

(24)

Because (24) holds true for $\forall k\in \overline {G(t-1)}$ and $k_{t+1}^{*} \in \overline {G(t)} \subset \overline {G(t-1)}$ in (20), we can rewrite (20) as

$$ \begin{array}{@{}rcl@{}} s^{\{k_{t+1}^{*}\}}_{b}s_{w}^{G(t)}-s_{b}^{G(t)}s^{\{k_{t+1}^{*}\}}_{w}\leq 0 \end{array} $$

(25)

Substituting (25) into (19), we have

$$ \begin{array}{@{}rcl@{}} J(G(t+1))-J(G(t))\leq 0 \end{array} $$

(26)

Consequently, we complete the proof of Theorem 1 by using the mathematical induction method.□