A recursive feature retention method for semi-supervised feature selection

Abstract

To deal with semi-supervised feature selection tasks, this paper presents a recursive feature retention (RFR) method based on a neighborhood discriminant index (NDI) method (a supervised feature selection method) and a forward iterative Laplacian score (FILS) method (an unsupervised method), where FILS is designed specially for RFR. The goal of RFR is to determine an optimal feature subset that has not only a high discriminant ability but also a strong ability to maintain the local structure of data. The discriminant ability of a feature is measured by NDI, and the ability of a feature to maintain the local structure of data is described by FILS. RFR compromises these two scores to give a balanced score for a feature. RFR iteratively selects a feature with the smallest balanced score and moves it into the current optimal feature subset. This paper also shows theoretical analysis to speed up iterations. Extensive experiments are conducted on toy and real-world data sets. Experimental results confirm that RFR can achieve a better performance compared with the state-of-the-art semi-supervised methods.

Appendices

Proof of Theorem 1

Proof

For a given non-empty set A, We need to prove that $J(A)\ge 0$ holds true for $1\le t\le n$. When $A \ne \emptyset$, its Laplacian score is described as

$$\begin{aligned} J(A)=\frac{trace\left( \widetilde{{\mathbf {Z}}}_{A}^T{\mathbf {L}} \widetilde{{\mathbf {Z}}}_{A}\right) }{trace\left( \widetilde{{\mathbf {Z}}}_{A}^T{\mathbf {D}}\widetilde{{\mathbf {Z}}}_{A}\right) } \end{aligned}$$

(19)

The numerator of J(A) can be rewritten as

$$\begin{aligned} trace\left( \widetilde{{\mathbf {Z}}}_{A}^T{\mathbf {L}}\widetilde{{\mathbf {Z}}}_{A}\right) = \sum _{f_m\in A}\widetilde{{\mathbf {z}}}_{f_m}^T{\mathbf {L}}\widetilde{{\mathbf {z}}}_{f_m} \end{aligned}$$

(20)

where $\widetilde{{\mathbf {z}}}_{f_m}$ is a column of $\widetilde{{\mathbf {Z}}}_{A}$. Because the Laplacian matrix ${\mathbf {L}}$ is symmetric and positive semi-definite, we have

$$\begin{aligned} \widetilde{{\mathbf {z}}}_{f_m}^T{\mathbf {L}}\widetilde{{\mathbf {z}}}_{f_m} \ge 0 \end{aligned}$$

(21)

which indicates that the numerator (20) of $J(A^*(t))$ is nonnegative, or

$$\begin{aligned} trace\left( \widetilde{{\mathbf {Z}}}_{A}^T{\mathbf {L}}\widetilde{{\mathbf {Z}}}_{A}\right) \ge 0 \end{aligned}$$

(22)

Similarly, the denominator of J(A) can be rewritten as

$$\begin{aligned} trace\left( \widetilde{{\mathbf {Z}}}_{A}^T{\mathbf {D}}\widetilde{{\mathbf {Z}}}_{A}\right) = \sum _{f_m\in A}\widetilde{{\mathbf {z}}}_{f_m}^T{\mathbf {D}}\widetilde{{\mathbf {z}}}_{f_m} \end{aligned}$$

(23)

Moreover, the matrix ${\mathbf {D}}$ is a diagonal matrix that is positive definite. Thus we have

$$\begin{aligned} trace\left( \widetilde{{\mathbf {Z}}}_{A}^T{\mathbf {D}}\widetilde{{\mathbf {Z}}}_{A}\right) > 0 \end{aligned}$$

(24)

By (22) and (24), when $A\ne \emptyset$ we can have the conclusion:

$$\begin{aligned} J(A)\ge 0 \end{aligned}$$

(25)

which completes the proof of Theorem 1. $\square$

Proof of Theorem 2

Proof

To prove the inequalities (13) in Theorem 2, we use the mathematical induction method.

When $t=1$, $A^*(0)=\emptyset$ and $A^*(1)\ne \emptyset$. Since $J(A^*(0))=-\infty$ and $J(A^*(1))\ge 0$ according to (11), $J(A^*(0))\le J(A^*(1))$ is true. For simplification, let

$$\begin{aligned} trace\left( \widetilde{{\mathbf {Z}}}_{A}^T{\mathbf {L}}\widetilde{{\mathbf {Z}}}_{A}\right) =\sum _{f_m\in A} a_{f_m},~~trace\left( \widetilde{{\mathbf {Z}}}_{A}^T{\mathbf {D}}\widetilde{{\mathbf {Z}}}_{A} \right) =\sum _{f_m\in A} b_{f_m} \end{aligned}$$

When $t=2$, the Laplacian score of a feature subset A(1) with $|A(1)|=1$ can be reduced as:

$$\begin{aligned} J(A^*(1))=\frac{a_1^*}{b_1^*}=\min _{f_k\in F}\frac{a_{f_k}}{b_{f_k}} \end{aligned}$$

(26)

where $a_1^*$ and $b_1^*$ are the corresponding $a_{f_m}$ and $b_{f_m}$ under the optimal solution, respectively.

$$\begin{aligned} J(A^*(2))=\frac{a_2^*}{b_2^*}=\frac{a_1^*+a_{p_1}}{b_1^*+b_{p_1}}=\min _{f_k\in \overline{A^*(1)}}\frac{a_1^*+a_{f_k}}{b_1^*+b_{f_k}} \end{aligned}$$

(27)

where $a_2^*=a_1^*+a_{p_1}$ and $b_2^*=b_1^*+b_{p_1}$, ${p_1}$ corresponds the optimal feature index in the second iteration. Combining (26) and (27), we have

$$\begin{aligned} J(A^*(1))-J(A^*(2))= & {} \frac{a_1^*}{b_1^*}-\frac{a_2^*}{b_2^*} \nonumber \\= & {} \frac{a_1^*}{b_1^*}-\frac{a_1^*+a_{p_1}}{b_1^*+b_{p_1}}\nonumber \\= & {} \frac{a_1^*(a_1^*+a_{p_1})-b_1^*(a_1^*+a_{p_1})}{b_1^*(b_1^*+b_{p_1})}\nonumber \\= & {} \frac{a_1^*b_{p_1}-b_1^*a_{p_1}}{b_1^*(b_1^*+b_{p_1})} \end{aligned}$$

(28)

According to (26), we know

$$\begin{aligned} \frac{a_1^*}{b_1^*}\le \frac{a_{p_1}}{b_{p_1}} \end{aligned}$$

(29)

Thus, we have

$$\begin{aligned} a_1^*b_{p_1}-b_1^*a_{p_1}\le 0 \end{aligned}$$

(30)

Considering $b_1^*(b_1^*+b_{p_1})>0$ and (30), we lead a conclusion that $J(A^*(1))\le J(A^*(2))$ is true.

Assume that $J(A^*(N-1))\le J(A^*(N))$ is true for $t=N<n$. Without loss of generality, let

$$\begin{aligned} J(A^*(N-1))=\frac{a_{N-1}^*}{b_{N-1}^*} \end{aligned}$$

(31)

and

$$\begin{aligned} J(A^*(N))=\frac{a_{N}^*}{b_{N}^*}=\frac{a_{N-1}^*+a_{p_{N-1}}}{b_{N-1}^*+b_{p_{N-1}}}= \min _{f_k\in \overline{A^*(N-1)}} \frac{a_{N-1}^*+a_{f_k}}{b_{N-1}^*+b_{f_k}} \end{aligned}$$

(32)

where ${p_{N-1}}$ is the optimal feature index in the N-th iteration. According to $J(A^*(N-1))\le J(A^*(N))$, (31) and (32), for $\forall f_k \in \overline{A^*(N-1)}$ we have

$$\begin{aligned} \frac{a_{N-1}^*}{b_{N-1}^*}\le \frac{a_{N-1}^*+a_{p_{N-1}}}{b_{N-1}^*+b_{p_{N-1}}}\le \frac{a_{N-1}^*+a_{f_k}}{b_{N-1}^*+b_{f_k}} \end{aligned}$$

(33)

Further, we have

$$\begin{aligned} \frac{a_{N-1}^*}{b_{N-1}^*} \le \frac{a_{N-1}^*+a_{p_{N-1}}}{b_{N-1}^*+b_{p_{N-1}}} \Rightarrow \frac{a_{N-1}^*}{b_{N-1}^*} \le \frac{a_{p_{N-1}}}{b_{p_{N-1}}} \Rightarrow \nonumber \\ a_{N-1}^*b_{p_{N-1}}-b_{N-1}^*a_{p_{N-1}} \le 0 \end{aligned}$$

(34)

$$\begin{aligned} \frac{a_{N-1}^*}{b_{N-1}^*}\le \frac{a_{N-1}^*+a_{f_k}}{b_{N-1}^*+b_{f_k}} \Rightarrow \frac{a_{N-1}^*}{b_{N-1}^*}\le \frac{a_{f_{k}}}{b_{f_{k}}} \Rightarrow \nonumber \\ a_{N-1}^*b_{f_{k}}-b_{N-1}^*a_{f_{k}}\le 0 \end{aligned}$$

(35)

$$\begin{aligned} \frac{a_{N-1}^*+a_{p_{N-1}}}{b_{N-1}^*+b_{p_{N-1}}} \le \frac{a_{N-1}^*+a_{f_k}}{b_{N-1}^* +b_{f_k}} \Rightarrow \nonumber \\ (a^*_{N-1}b_{f_k}-b^*_{N-1}a_{f_k})+ \left( a_{p_{N-1}}b_{f_k}-b_{p_{N-1}}a_{f_k}\right) \le \nonumber \\ \left( a^*_{N-1}b_{{p_{N-1}}}-b^*_{N-1}a_{{p_{N-1}}} \right) \end{aligned}$$

(36)

where $f_k \in \overline{A^*(N-1)}$.

When $t=N+1 \le n$, we want to prove that $J(A^*(N))\le J(A^*(N+1))$ is true. Let

$$\begin{aligned} J(A^*(N+1))=\frac{a_{N+1}^*}{b_{N+1}^*}=\frac{a_{N}^*+a_{p_{N}}}{b_{N}^*+b_{p_{N}}} \end{aligned}$$

(37)

where ${p_{N}}$ is the optimal feature index in the $(N+1)$-th iteration. We compute $J(A^*(N))-J(A^*(N+1))$, and have

$$\begin{aligned}&J(A^*(N))-J(A^*(N+1)) \nonumber \\= & {} \frac{a_{N}^*}{b_{N}^*}-\frac{a_{N+1}^*}{b_{N+1}^*} \nonumber \\= & {} \frac{a_{N}^*}{b_{N}^*}-\frac{a_{N}^*+a_{p_{N}}}{b_{N}^*+b_{p_{N}}}\nonumber \\= & {} \frac{a_{N}^*(a_{N}^*+a_{p_{N}})-b_{N}^*(a_{N}^*+a_{p_{N}})}{b_{N}^*(b_{N}^*+b_{p_{N}})}\nonumber \\= & {} \frac{a_{N}^*b_{p_{N}}-b_{N}^*a_{p_{N}}}{b_{N}^*(b_{N}^*+b_{p_{N}})}\nonumber \\= & {} \frac{\left( a_{N-1}^*b_{p_{N}}-b_{N-1}^*a_{p_{N}}\right) +\left( a_{p_{N-1}}b_{p_{N}} -b_{p_{N-1}}a_{p_{N}}\right) }{b_{N}^*(b_{N}^*+b_{p_{N}})} \end{aligned}$$

(38)

Substituting (36) into the last equation in (38) and replacing $f_k$ with $p_{N}$ owing to the arbitrariness of $f_k$, we have

$$\begin{aligned} J(A^*(N))-J(A^*(N+1)) \le \frac{\left( a^*_{N-1}b_{{p_{N-1}}}-b^*_{N-1}a_{{p_{N-1}}}\right) }{b_{N}^*(b_{N}^* +b_{p_{N}})}\le 0 \end{aligned}$$

(39)

which shows that $J(A^*(N))\le J(A^*(N+1))$ is true when $t=N+1$.

Consequently, by the Principle of Induction, $J(A^*(t-1))\le J(A^*(t))$ for $1 \le t\le n$. $\square$

Proof of Theorem 3

Proof

Note that the set $|B|=1$. We follow the notations in the proof procedure of Theorem 2. In the t-th iteration, assume that

$$\begin{aligned} J(A^*(t))=\frac{a_{t}^*}{b_{t}^*} \end{aligned}$$

(40)

For the $(t+1)$-th iteration, we have

$$\begin{aligned} J(A^*(t+1))=\frac{a_{t+1}^*}{b_{t+1}^*}=\frac{a_{t}^*+a_{p_{t}}}{b_{t}^*+b_{p_{t}}} \end{aligned}$$

(41)

where ${p_{t}}$ is the optimal feature index in the $(t+1)$-th iteration, or $A^*(t+1)-A^*(t)=B={f_{p_t}}$.

According to Theorem 2, we know that $J(A^*(t))-J(A^*(t+1))\le 0$, and have

$$\begin{aligned} J(A^*(t))-J(A^*(t+1))= & {} \frac{a_{t}^*b_{p_{t}}-b_{t}^*a_{p_{t}}}{b_{t}^*(b_{t}^*+b_{p_{t}})} \end{aligned}$$

(42)

Thus, according to $a_{t}^*b_{p_{t}}-b_{t}^*a_{p_{t}}\le 0$, we have

$$\begin{aligned} \frac{a_{t}^*}{b_{t}^*}\le \frac{a_{p_{t}}}{b_{p_{t}}} \end{aligned}$$

(43)

Since $a_{p_{t}}=\widetilde{{\mathbf {z}}}_{f_{p_{t}}}^T{\mathbf {L}} \widetilde{{\mathbf {z}}}_{f_{p_{t}}}=\widetilde{{\mathbf {Z}}}^T_B{\mathbf {L}} \widetilde{{\mathbf {Z}}}^T_B$, and $b_{p_{t}}=\widetilde{{\mathbf {z}}}_{f_{p_{t}}}^T{\mathbf {D}} \widetilde{{\mathbf {z}}}_{f_{p_{t}}}=\widetilde{{\mathbf {Z}}}^T_B{\mathbf {D}} \widetilde{{\mathbf {Z}}}^T_B$, (43) can be rewritten as

$$\begin{aligned} J(A^*(t))\le \frac{\widetilde{{\mathbf {Z}}}^T_B{\mathbf {L}}\widetilde{{\mathbf {Z}}}^T_B}{\widetilde{{\mathbf {Z}}}^T_B{\mathbf {D}}\widetilde{{\mathbf {Z}}}^T_B} \end{aligned}$$

(44)

which completes the proof of Theorem 3. $\square$

Proof of Theorem 5

Proof

According to the definition of neighborhood relation, we have

$$\begin{aligned} R_A^{\varepsilon }=\left\{ \left( \mathbf{x }_i,\mathbf{x }_j\right) |\Delta ^A \left( \mathbf{x }_i,\mathbf{x }_j\right) \le \varepsilon ,\left( \mathbf{x }_i,y_i\right) ,\left( \mathbf{x }_j,y_j\right) \in X_L\right\} \end{aligned}$$

(45)

and

$$\begin{aligned} R_{A^k}^{\varepsilon }=\left\{ \left( {\mathbf {x}}_i,{\mathbf {x}}_j\right) |\Delta ^{A^k} \left( \mathbf{x }_i,\mathbf{x }_j\right) \le \varepsilon ,\left( \mathbf{x }_i,y_i\right) ,\left( \mathbf{x }_j,y_j\right) \in X_L\right\} \end{aligned}$$

(46)

On the basis of $\Delta ^{A}\left( \mathbf{x }_i,\mathbf{x }_j\right)$, the distance function $\Delta ^{A^k}\left( \mathbf{x }_i,\mathbf{x }_j\right)$ can be rewritten as

$$\begin{aligned} \Delta ^{A^k}\left( \mathbf{x }_i,\mathbf{x }_j\right) =&\max _{f_q\in A^k} {\left| x_{iq}-x_{jq}\right| } \nonumber \\ =&\max \left( \Delta ^{A}\left( \mathbf{x }_i,\mathbf{x }_j\right) ,\left| x_{ik}-x_{jk} \right| \right) \end{aligned}$$

(47)

$\forall \left( {\mathbf {x}}_i,{\mathbf {x}}_j\right) \in R_{A^k}^{\varepsilon }$, we have $\Delta ^{A^k}\left( \mathbf{x }_i,\mathbf{x }_j\right) \le \varepsilon$. According to (47), we know that $\Delta ^{A}\left( \mathbf{x }_i,\mathbf{x }_j\right) \le \varepsilon$ holds true. Thus, we have $\left( {\mathbf {x}}_i,{\mathbf {x}}_j\right) \in R_{A}^{\varepsilon }$ in terms of (45). In other words, $R_{A^k}^{\varepsilon } \subseteq R_{A}^{\varepsilon }$, which completes the proof of Theorem 5. $\square$

Proof of Theorem 6

Proof

To prove the rules in Theorem 6, we can use the relationship between $\Delta ^{A^k}\left( \mathbf{x }_i,\mathbf{x }_j\right)$ and $\Delta ^{A}\left( \mathbf{x }_i,\mathbf{x }_j\right)$:

$$\begin{aligned} \Delta ^{A^k}\left( \mathbf{x }_i,\mathbf{x }_j\right) = \max \left( \Delta ^{A}\left( \mathbf{x }_i,\mathbf{x }_j\right) ,\left| x_{ik} -x_{jk}\right| \right) \end{aligned}$$

(48)

For the rule (1), we have

$$\begin{aligned}&\forall \left( {\mathbf {x}}_i,{\mathbf {x}}_j\right) \notin R_{A}^{\varepsilon } \Leftrightarrow \Delta ^{A}\left( \mathbf{x }_i,\mathbf{x }_j\right)>\varepsilon \\&\Rightarrow \Delta ^{A^k}\left( \mathbf{x }_i,\mathbf{x }_j\right) >\varepsilon \Leftrightarrow \left( {\mathbf {x}}_i,{\mathbf {x}}_j\right) \notin R_{A^k}^{\varepsilon } \end{aligned}$$

For the rule (2), we have

$$\begin{aligned}&\forall \left( {\mathbf {x}}_i,{\mathbf {x}}_j\right) \in R_{A}^{\varepsilon } \wedge \left| {x}_{ik}-x_{jk}\right|>\varepsilon \Leftrightarrow \Delta ^{A}\left( \mathbf{x }_i,\mathbf{x }_j\right) \le \varepsilon \wedge \left| {x}_{ik}-x_{jk}\right|>\varepsilon \\&\Rightarrow \Delta ^{A^k}\left( \mathbf{x }_i,\mathbf{x }_j\right) >\varepsilon \Leftrightarrow \left( {\mathbf {x}}_i,{\mathbf {x}}_j\right) \notin R_{A^k}^{\varepsilon } \end{aligned}$$

For the rule (3), we have

$$\begin{aligned}&\forall \left( {\mathbf {x}}_i,{\mathbf {x}}_j\right) \in R_{A}^{\varepsilon } \wedge \left| {x}_{ik}-x_{jk}\right| \le \varepsilon \Leftrightarrow \Delta ^{A}\left( \mathbf{x }_i,\mathbf{x }_j\right) \le \varepsilon \wedge \left| {x}_{ik}-x_{jk}\right| \le \varepsilon \\&\Rightarrow \Delta ^{A^k}\left( \mathbf{x }_i,\mathbf{x }_j\right) \le \varepsilon \Leftrightarrow \left( {\mathbf {x}}_i,{\mathbf {x}}_j\right) \in R_{A^k}^{\varepsilon } \end{aligned}$$

In summary, the three rules hold true. This completes the proof of Theorem 6. $\square$