Random walk-based fuzzy linear discriminant analysis for dimensionality reduction

Abstract

Dealing with high-dimensional data has always been a major problem with the research of pattern recognition and machine learning, and linear discriminant analysis (LDA) is one of the most popular methods for dimensionality reduction. However, it suffers from the problem of being too sensitive to outliers. Hence to solve this problem, fuzzy membership can be introduced to enhance the performance of algorithms by reducing the effects of outliers. In this paper, we analyze the existing fuzzy strategies and propose a new effective one based on Markov random walks. The new fuzzy strategy can maintain high consistency of local and global discriminative information and preserve statistical properties of dataset. In addition, based on the proposed fuzzy strategy, we then derive an efficient fuzzy LDA algorithm by incorporating the fuzzy membership into learning. Theoretical analysis and extensive simulations show the effectiveness of our algorithm. The presented results demonstrate that our proposed algorithm can achieve significantly improved results compared with other existing algorithms.

Appendix

1.1 Proof of Corollary 1

According to Eq. (12), we have

$$ \begin{aligned} ( {T^{\rm T} T} )_{ij} &= \sum\limits_{k = 1}^{c} {\left( {\frac{{w_{ki} }}{{\sqrt {F_{kk} } }} - \frac{{\sqrt {F_{kk} } }}{l}} \right)\left( {\frac{{w_{kj} }}{{\sqrt {F_{kk} } }} - \frac{{\sqrt {F_{kk} } }}{l}} \right)} \\ &= \sum\limits_{k = 1}^{c} {\frac{{w_{ki} w_{kj} }}{{F_{kk} }}} - \sum\limits_{k = 1}^{c} {\frac{{w_{ki} }}{l}} - \sum\limits_{k = 1}^{c} {\frac{{w_{kj} }}{l}} + \sum\limits_{k = 1}^{c} {\frac{{F_{kk} }}{{l^{2} }}} \\ &= \sum\limits_{k = 1}^{c} {\frac{{w_{ki} w_{kj} }}{{F_{kk} }}} - \frac{1}{l} = - ( {\widetilde{{A_{\rm b} }}} )_{ij}. \end{aligned} $$

(26)

According to Eq. (8), we have

$$ \begin{aligned} ( {\widetilde{{D_{\rm b} }}} )_{ii} = \sum\limits_{j = 1}^{l} {( {\widetilde{{A_{\rm b} }}} )_{ij} } &= \sum\limits_{j = 1}^{l} {\left( {\frac{1}{l} - \sum\limits_{k = 1}^{c} {\frac{{w_{ki} w_{kj} }}{{F_{kk} }}} } \right)} \\ &= 1 - \sum\limits_{k = 1}^{c} {\frac{1}{{F_{kk} }}\sum\limits_{j = 1}^{l} {w_{ki} w_{kj} } } \\ &= 1 - \sum\limits_{k = 1}^{c} {w_{ki} } = 1 - 1 = 0 \\ \end{aligned} $$

(27)

The second equality holds as $\sum\nolimits_{j=1}^{l} w_{kj}= F_{kk}$ and the third equality holds as $\sum\nolimits_{k=1}^{c} w_{ki}= 1.$ We then have $( {\widetilde{{L_{\rm b} }}} )_{ij} = ( {\widetilde{{D_{\rm b} }}} )_{ii} - ( {\widetilde{{A_{\rm b} }}} )_{ij} = - ( {\widetilde{{A_{\rm b} }}})_{ij} ,$ hence we prove $T^{\rm T} T = \widetilde{{L_{\rm b} }}.$

1.2 Proof of Corollary 2

We prove Corollary 2 using a recursive algorithm. According to Eq. (18) and W(0) = Y, have

$$ \begin{aligned} \sum\limits_{i = 1}^{c} {w_{ij} ( 1 )} &= \alpha \sum\limits_{i = 1}^{c} {\sum\limits_{{x_{k} \in N_{k} ( {x_{j} } )}} {y_{ik} s_{kj} } } + ( {1 - \alpha } )\sum\limits_{i = 1}^{c} {y_{ij} } \\ &= \alpha \sum\limits_{{x_{k} \in N_{k} ( {x_{i} } )}} {s_{kj} \sum\limits_{i = 1}^{c} {y_{ik} } } + ( {1 - \alpha } )\sum\limits_{i = 1}^{c} {y_{ij} } \\ &= \alpha \sum\limits_{{x_{k} \in N_{k} ( {x_{i} })}} {s_{kj} } + ( {1 - \alpha } ) = \alpha + ( {1 - \alpha } ) = 1 \end{aligned} $$

(28)

The third equality holds as $\sum\nolimits_{i=1}^{c} y_{ik}= 1$∑ ^c _i=1 y _ik  = 1 and the fourth equality holds as $\sum\nolimits_{{x_{k} \in N_{k} \left( {x_{i} } \right)}} {s_{kj} } = 1$. Hence, Eq. (28) indicates that the sum of each column of W(1) is equivalent to 1. We next assume that the sum of each column of W(t) is equivalent to 1, i.e. $\sum\nolimits_{i=1}^{c} w_{ij} (t) = 1$ for any iteration t, we then have

$$ \begin{aligned} \sum\limits_{i = 1}^{c} {w_{ij} ( {t + 1})} &= \alpha \sum\limits_{i = 1}^{c} {\sum\limits_{{x_{k} \in N_{k} ( {x_{i} } )}} {w_{ik} ( t )} } s_{kj} + ( {1 - \alpha } )\sum\limits_{i = 1}^{c} {y_{ij} } \\ &= \alpha \sum\limits_{{x_{k} \in N_{k} ( {x_{i} } )}} {s_{kj} \sum\limits_{i = 1}^{c} {w_{ik} ( t )} } + ( {1 - \alpha } )\sum\limits_{i = 1}^{c} {y_{ij} } \\ &= \alpha \sum\limits_{{x_{k} \in N_{k} ( {x_{i} } )}} {s_{kj} } + ( {1 - \alpha } ) = \alpha + ( {1 - \alpha } ) = 1 \end{aligned} $$

(29)

This indicates that the sum of each column of W(t + 1) is also equivalent to 1. Thus, we prove that $\sum\nolimits_{i = 1}^{c} {w_{ij} } = \sum\nolimits_{i = 1}^{c} {\mathop {\lim }\limits_{t \to \infty } w_{ij} ( t )} = 1.$

1.3 Proof of Theorem 1

• 1. Computing V ^* _F via eigen-decomposition (Ye 2005)

Let t be the rank of $\widetilde{{S_{\rm t} }},$ by performing SVD to $\widetilde{{S_{\rm t} }},$ we have

$$ \widetilde{{S_{\rm t} }} = U\left( {\begin{array}{*{20}l} {\Upsigma_{t}^{2} } & 0 \\ 0 & 0 \\ \end{array} } \right)U^{\rm T} , $$

(30)

where U is an orthogonal matrix, Σ ² _t is a diagonal matrix with rank $t.$ Let U = [U ₁, U ₂] be a partition of U such that $U_{1} \in \mathbb{R}^{d \times t}$ and $U_{2} \in \mathbb{R}^{d \times (d - t)} ,$ where U ₂ lies in the null space of $\widetilde{{S_{\rm t} }}$ satisfying $U_{{_{2} }}^{\rm T} \widetilde{{S_{\rm t} }}U_{2} = 0,$ we then have $S_{\rm t} = U_{1} \sum_{\rm t}^{2} U_{1}^{\rm T} .$ Since $\widetilde{{S_{\rm t} }} = \widetilde{{S_{\rm b} }} + \widetilde{{S_{\rm w} }},$ we have

$$ U^{\rm T} \widetilde{{S_{\rm b} }}U = \left( {\begin{array}{*{20}l} {U_{1}^{\rm T} \widetilde{{S_{\rm b} }}U_{1} } & 0 \\ 0 & 0 \\ \end{array} } \right),U^{\rm T} \widetilde{{S_{\rm w} }}U = \left( {\begin{array}{*{20}l} {U_{1}^{\rm T} \widetilde{{S_{\rm w} }}U_{1} } & 0 \\ 0 & 0 \\ \end{array} } \right) $$

(31)

From Eqs. (30, 31), it follows

$$ I_{\rm t} = \sum_{\rm t}^{ - 1} U_{1}^{T} \widetilde{{S_{\rm t} }}U_{1} \sum_{\rm t}^{ - 1} = \sum_{\rm t}^{ - 1} U_{1}^{\rm T} \widetilde{{S_{\rm b} }}U_{1} \sum_{\rm t}^{ - 1} + \sum_{\rm t}^{ - 1} U_{1}^{T} \widetilde{{S_{\rm w} }}U_{1} \sum_{\rm t}^{ - 1} , $$

(32)

where $I_{\rm t} \in \mathbb{R}^{t \times t}$ is an identity matrix. Recall that $\widetilde{{S_{\rm b} }} = \widetilde{{H_{\rm b} }}\widetilde{{H_{\rm b} }}^{\rm T} ,$ if we let $G = \sum_{\rm t}^{ - 1} U_{1}^{\rm T} \widetilde{{H_{\rm b} }}$ and its SVD be $G = P\sum_{\rm b} Q,$ where $P \in \mathbb{R}^{t \times t}$ and $Q \in \mathbb{R}^{t \times c}$ are two orthogonal matrixes and $\sum_{b} \in \mathbb{R}^{t \times t}$ is a diagonal matrix, we then have

$$ \sum_{t}^{ - 1} U_{1}^{\rm T} \widetilde{{S_{b} }}U_{1} \sum_{t}^{ - 1} = GG^{\rm T} = P\sum_{b}^{2} P^{\rm T} . $$

(33)

Therefore, according to Eqs. (30, 31, 33), we rewrite $V_{F}^{*} = \widetilde{{S_{\rm t} }}^{ - 1} \widetilde{{S_{\rm b} }}$ as

$$ \begin{aligned} \widetilde{{S_{\rm t} }}^{ - 1} \widetilde{{S_{\rm b} }} &= U_{1} \left( {\begin{array}{*{20}l} {\sum_{t}^{ - 1} \sum_{t}^{ - 1} } & 0 \\ 0 & 0 \\ \end{array} } \right)U^{\rm T} \widetilde{{S_{\rm b} }}U\left( {\begin{array}{*{20}l} {\sum_{t}^{ - 1} \sum_{\rm t} } & 0 \\ 0 & 0 \\ \end{array} } \right)U^{\rm T} \\ &= U\left( {\begin{array}{*{20}l} {\sum_{t}^{ - 1} } & 0 \\ 0 & 0 \\ \end{array} } \right)P\sum_{b}^{2} P^{\rm T} \left( {\begin{array}{*{20}l} {\sum_{t} } & 0 \\ 0 & 0 \\ \end{array} } \right)U^{\rm T} \\ & = U\left( {\begin{array}{*{20}l} {\sum_{t}^{ - 1} P} & 0 \\ 0 & {I^{D - t} } \\ \end{array} } \right)\left( {\begin{array}{*{20}l} {\sum_{b}^{2} } & 0 \\ 0 & 0 \\ \end{array} } \right)\left( {\begin{array}{*{20}l} {P^{\rm T} \sum_{t} } & 0 \\ 0 & {I^{D - t} } \\ \end{array} } \right)U^{\rm T} \end{aligned} $$

(34)

The first equality holds as Eq. (30), the second equality holds as Eq. (33). From Eq. (34), if we let $V_{{^{F} }}^{*} = U_{1} \sum_{t}^{ - 1} P,$ we have $\widetilde{{S_{\rm t} }}^{ - 1} \widetilde{{S_{\rm b} }}V_{{^{F} }}^{*} = \sum_{\rm b}^{2} V_{{^{F} }}^{*} ,$ which indicate the column vectors of $V_{{^{F} }}^{*}$ are eigenvectors of $\widetilde{{S_{\rm t} }}^{ - 1} \widetilde{{S_{\rm b} }}.$

• 2. Equivalence relationship to least square

Recall $V_{\rm LS}^{*}$ in Eq. (11), it can be rewritten as

$$ \begin{aligned} \widetilde{{S_{\rm t} }}^{ - 1} \widetilde{{H_{\rm b} }} &= U\left( {\begin{array}{*{20}l} {\sum_{t}^{ - 1} \sum_{t}^{ - 1} } & 0 \\ 0 & 0 \\ \end{array} } \right)U^{\rm T} \widetilde{{H_{\rm b} }} \\ & = U_{1} \sum_{t}^{ - 1} \left( {\sum_{t}^{ - 1} U_{1}^{\rm T} \widetilde{{H_{\rm b} }}} \right) \\ & = U_{1} \sum_{t}^{ - 1} G \\ & = U_{1} \sum_{t}^{ - 1} P\sum_{\rm b} Q^{\rm T} \\ & = V_{F}^{*} \sum_{\rm b} Q^{\rm T} \end{aligned} $$

(35)

From Eq. (35), we can neglect Q as it is orthogonal. Thus, the main difference between $V_{F}^{*}$ and $V_{\rm LS}^{*}$ is the diagonal matrix $\sum_{b} .$ We next show that given the condition in Eq. (13), ∑ _b is an identity matrix hence resulting in $V_{\rm LS}^{*} = V_{F}^{*} .$ Let $H \in \mathbb{R}^{D \times D}$ be a non-degenerate matrix defined as:

$$ H = U\left( {\begin{array}{*{20}l} {\sum_{t}^{ - 1} P} & 0 \\ 0 & {I_{D - t} } \\ \end{array} } \right) $$

(36)

According to Eq. (31) (32) and $\widetilde{{S_{\rm w} }} = \widetilde{{S_{\rm t} }} - \widetilde{{S_{\rm b} }},$ we have

$$ H^{T} \widetilde{{S_{\rm t} }}H = \left( {\begin{array}{*{20}l} {I_{\rm t} } & 0 \\ 0 & 0 \\ \end{array} } \right),H^{\rm T} \widetilde{{S_{\rm w} }}H = \left( {\begin{array}{*{20}l} {\sum_{w}^{2} } & 0 \\ 0 & 0 \\ \end{array} } \right),H^{\rm T} \widetilde{{S_{\rm b} }}H = \left( {\begin{array}{*{20}l} {\sum_{b}^{2} } & 0 \\ 0 & 0 \\ \end{array} } \right) $$

(37)

where ∑ ² _b  = diag(σ ²₁ , σ ²₂ , …, σ ² _t , 0, …, 0) and ∑ ² _w  = diag(τ ²₁ , τ ²₂ , …, τ ² _t , 0, …, 0) are two diagonal matrixes satisfying σ ² _i  + τ ² _i  = 1, ∀i. This indicates that there is at least one of σ _i and τ _i to be nonzero, ∀i. Since rank(A) + rank(B) ≥ rank(A + B) (Hull 1994), we have ${\text{rank}}( {\widetilde{{S_{\rm b} }}}) + {\text{rank}} ( {\widetilde{{S_{\rm w} }}}) \ge {\text{rank}}( {\widetilde{{S_{\rm t} }}})$ According to the Sylvester’s law of interia (Hull 1994), it follows rank(∑ ² _b ) + rank(∑ ² _w ) ≥ rank(I _t ). Let b be the rank of ∑ ² _b and assume rank(∑ ² _b ) + rank(∑ ² _w ) = rank(I _t ) + s, to satisfy this rank equality, we have

$$ \begin{gathered} 1 = \sigma_{1}^{2} = \sigma_{2}^{2} = \cdots = \sigma_{b - s}^{2} > \sigma_{b - s + 1}^{2} > \cdots \sigma_{b}^{2} > \sigma_{b + 1}^{2} = \cdots \sigma_{t}^{2} = 0 \hfill \\ 0 = \tau_{1}^{2} = \tau_{2}^{2} = \cdots \tau_{b - s}^{2} < \tau_{b - s + 1}^{2} < \cdots \tau_{b}^{2} < \tau_{b + 1}^{2} = \cdots \tau_{t}^{2} = 1 \hfill \\ \end{gathered} $$

(38)

Since C1 holds, we have s = 0 and 1 = σ ²₁  = σ ²₂  = ··· = σ ² _b  > σ ² _b+1  = ··· = σ ² _t  = 0, which indicates that ∑ _b is an identity matrix.

1.4 Appendix D

• 1. Computing V ^* _F via two-stage approach

In the two-stage approach, we first solve a least square problem by regressing X on T, i.e. projecting the original high-dimensional dataset into a low-dimensional subspace, we then calculate a auxiliary matrix M ∈ R ^d×d and its SVD. Finally, the optimal projection matrix can be obtained from the SVD of M. Since the size of M is very small, the cost for calculating the SVD of M is relatively low. The basic steps of two-stage approach are listed as follows:

• 1. Solve the least square problem $\mathop {\min }\limits_{V} \| {T^{T} - X^{T} V} \|_{F}^{2}$ and obtain the optimal solution V ^*_LS .

• 2. Let $\widetilde{X} = V_{\rm LS}^{*{\rm T}} X$ and calculate the auxiliary matrix as $M = V_{\rm LS}^{*{\rm T}} XT^{\rm T} .$

• 3. Perform SVD to M as M = U _M Σ _M U ^T _M and obtain $V_{M}^{*} = U_{M} \Upsigma_{M}^{ - 1/2} .$

• 4. The optimal solution can be given by $V_{\rm T}^{*} = V_{\rm LS}^{*} V_{M}^{*} .$

• 2. Equivalent relationship

We next prove the optimal solution V ^* _T obtained by two-stage approach is equivalent to that in Eq. (34). By solving least square problem in Eq. (11), we have V ^* _LS  = (XX ^T)⁻¹ XT ^T. Hence, $\widetilde{X} = V_{\rm LS}^{*T} X = TX^{\rm T} (XX^{\rm T} )^{ - 1} X$ The auxiliary matrix M can then be given by

$$ M = \widetilde{X}Y^{\rm T} = TX^{\rm T} (XX^{\rm T} )^{ - 1} XT^{\rm T} = \widetilde{{H_{\rm b} }}^{\rm T} U_{1} \sum_{t}^{ - 1} \sum_{t}^{ - 1} U_{1}^{\rm T} \widetilde{{H_{\rm b} }}. $$

(39)

The third equation holds as $\widetilde{{H_{\rm b} }} = XT^{\rm T}$ and XX ^T = S _t  = U ₁ ∑ ² _t U ^T₁ . Since $G = \sum_{t}^{ - 1} U_{1}^{\rm T} \widetilde{{H_{\rm b} }}$ and its SVD is G = P ∑ _b Q ^T, we have M = G ^T G = Q ∑ ² _b Q ^T. This indicates that Q ∑ ² _b Q ^T is a SVD of M, we thus have V ^* _M  = Q ∑ ⁻¹ _b and the optimal solution of two-stage approach can be given by:

$$ \begin{aligned} V_{\rm T}^{*} &= V_{\rm LS}^{*} V_{M}^{*} = (XX^{\rm T} )^{ - 1} XY^{\rm T} Q\sum_{b}^{ - 1} \\ & = U_{1} \sum_{1}^{ - 1} \left( {\sum_{1}^{ - 1} U_{1}^{\rm T} \widetilde{{H_{\rm b} }}} \right)Q\sum_{b}^{ - 1} \\ & = U_{1} \sum_{1}^{ - 1} P\sum_{b} Q^{\rm T} Q\sum_{b}^{ - 1} \\ & = U_{1} \sum_{1}^{ - 1} P, \\ \end{aligned} $$

(40)

which is equivalent to $V_{{^{F} }}^{*}$ in Eq. (34).