Filter-based unsupervised feature selection using Hilbert–Schmidt independence criterion

Abstract

Feature selection is a fundamental preprocess before performing actual learning; especially in unsupervised manner where the data are unlabeled. Essentially, when there are too many features in the problem, dimensionality reduction through discarding weak features is highly desirable. In this paper, we present a framework for unsupervised feature selection based on dependency maximization between the samples similarity matrices before and after deleting a feature. In this regard, a novel estimation of Hilbert–Schmidt independence criterion (HSIC), more appropriate for high-dimensional data with small sample size, is introduced. Its key idea is that by eliminating the redundant features and/or those have high inter-relevancy, the pairwise samples similarity is not affected seriously. Also, to handle the diagonally dominant matrices, a heuristic trick is used in order to reduce the dynamic range of matrix values. In order to speed up the proposed scheme, the gap statistic and k-means clustering methods are also employed. To assess the performance of our method, some experiments on benchmark datasets are conducted. The obtained results confirm the efficiency of our unsupervised feature selection scheme.

Appendix 1

1.1 Applying SP-BAHSIC, SPC-BAHSIC, SP-FOHSIC and SPC-FOHSIC on synthetic data

Herein, we explain the steps of our proposed methods on two synthetic data. First, we run SP-BAHSIC on dataset ${X_G}$ with $m=4$ samples and $n=4$ features, called $G=\left\{ {A,~B,~C,D} \right\}$.

$${X_G}=\left[ {\begin{array}{*{20}{l}} 1&3&4&2 \\ 3&6&5&4 \\ 5&{10}&4&3 \\ 2&5&4&4 \end{array}} \right]$$

Our aim is to find ${n^\prime }=2$ most informative features. By using RBF kernel function [29] as $\phi (.)$, the samples similarity matrix $K_{{\{ A,B,C,D\} }}^{\phi }$ is computed as:

$$K_{{\{ A,B,C,D\} }}^{\phi }=\left[ {\begin{array}{*{20}{l}} 1& \quad {0.70}& \quad {0.27}& \quad {0.83} \\ {0.70}& \quad 1& \quad {0.64}& \quad {0.94} \\ {0.27}& \quad {0.64}& \quad 1& \quad {0.50} \\ {0.83}& \quad {0.94}& \quad {0.50}& \quad 1 \end{array}} \right]$$

Similarly, using RBF kernel for $\psi (.)$, the samples similarity matrices $K_{{\{ B,C,D\} }}^{\psi }$, $K_{{\{ A,C,D\} }}^{\psi }$, $K_{{\{ A,B,D\} }}^{\psi }$ and $K_{{\{ A,B,C\} }}^{\psi }$ are obtained.

$$K_{{\{ B,C,D\} }}^{\psi }=\left[ {\begin{array}{*{20}{l}} 1&{0.93}&{0.78}&{0.96} \\ {0.93}&1&{0.91}&{0.99} \\ {0.78}&{0.91}&1&{0.88} \\ {0.96}&{0.99}&{0.88}&1 \end{array}} \right],$$

$$K_{{\{ A,C,D\} }}^{\psi }=\left[ {\begin{array}{*{20}{l}} 1&{0.96}&{0.92}&{0.97} \\ {0.96}&1&{0.97}&{0.99} \\ {0.92}&{0.97}&1&{0.95} \\ {0.97}&{0.99}&{0.95}&1 \end{array}} \right],$$

$$K_{{\{ A,B,D\} }}^{\psi }=\left[ {\begin{array}{*{20}{l}} 1&{0.92}&{0.72}&{0.96} \\ {0.92}&1&{0.90}&{0.99} \\ {0.72}&{0.90}&1&{0.84} \\ {0.96}&{0.99}&{0.84}&1 \end{array}} \right],$$

$$K_{{\{ A,B,C\} }}^{\psi }=\left[ {\begin{array}{*{20}{l}} 1& \quad {0.93}& \quad {0.72}& \quad {0.97} \\ {0.93}& \quad 1& \quad {0.90}& \quad {0.98} \\ {0.72}& \quad {0.90}& \quad 1& \quad {0.84} \\ {0.97}& \quad {0.98}& \quad {0.84}& \quad 1 \end{array}} \right]$$

Using (14), HSIC₂ is computed between $K_{{\{ A,B,C,D\} }}^{\phi }$ and each of $K_{{\{ B,C,D\} }}^{\psi }$, $K_{{\{ A,C,D\} }}^{\psi }$, $K_{{\{ A,B,D\} }}^{\psi }$ and $K_{{\{ A,B,C\} }}^{\psi }$. ${H_1}$ is achieved as:

$${H_1}=\left[ {\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} {0.0073}&{0.0027} \end{array}}&{\begin{array}{*{20}{c}} {0.0094}&{0.0093} \end{array}} \end{array}} \right]$$

According to ${H_1}$, the HSIC₂ measure between $K_{{\{ A,B,C,D\} }}^{\phi }$ and $K_{{\{ A,B,D\} }}^{\psi }$ is maximum. So, among 4 features, feature $C$ is more suitable for elimination. In the second phase, the samples similarity matrices $K_{{\{ B,D\} }}^{\psi }$, $K_{{\{ A,D\} }}^{\psi }$ and $K_{{\{ A,B\} }}^{\psi }$ are computed.

$$\begin{aligned} K_{{\{ B,D\} }}^{\psi } & =\left[ {\begin{array}{*{20}{l}} 1& \quad {0.94}& \quad {0.78}& \quad {0.96} \\ {0.94}& \quad 1& \quad {0.92}& \quad {0.99} \\ {0.78}& \quad {0.92}& \quad 1& \quad {0.88} \\ {0.96}& \quad {0.99}& \quad {0.88}& \quad 1 \end{array}} \right], \\ K_{{\{ A,D\} }}^{\psi } & =\left[ {\begin{array}{*{20}{l}} 1& \quad {0.96}& \quad {0.92}& \quad {0.97} \\ {0.96}& \quad 1& \quad {0.97}& \quad {0.99} \\ {0.92}& \quad {0.97}& \quad 1& \quad {0.95} \\ {0.97}& \quad {0.99}& \quad {0.95}& \quad 1 \end{array}} \right], \\ K_{{\{ A,B\} }}^{\psi } & =\left[ {\begin{array}{*{20}{l}} 1& \quad {0.94}& \quad {0.72}& \quad {0.97} \\ {0.94}& \quad 1& \quad {0.90}& \quad {0.99} \\ {0.72}& \quad {0.90}& \quad 1& \quad {0.84} \\ {0.97}& \quad {0.99}& \quad {0.84}& \quad 1 \end{array}} \right] \\ \end{aligned}$$

Also, HSIC₂ value between $K_{{\{ A,B,C,D\} }}^{\phi }$ and each of them is stored in ${H_2}$.

$${H_2}=\left[ {\begin{array}{*{20}{c}} {0.0072}&{0.0026}&{0.0093} \end{array}} \right]$$

From HSIC₂ values in ${H_2}$, the next candidate for elimination is feature $D$. Thus, features $A$ and $B$ are returned by SP-BAHSIC as the most informative features.

As another example, now SPC-BAHSIC is run on dataset ${X_G}$ with $m=4$ samples and $n=6$ features, named $G=\left\{ {A,~B,~C,D,E,F} \right\}$ in order to find $n'=2$ most informative features.

$${X_G}=\left[ {\begin{array}{*{20}{l}} 1&3&4&2&4&8 \\ 3&6&5&4&2&1 \\ 5&9&4&3&7&6 \\ 2&5&4&4&3&2 \end{array}} \right]$$

First, we estimate the number of clusters for these $n=6$ features using gap statistic method which results in $l=4$ features. Next, the clusters of features are found using k-means method. From each cluster, one feature is selected in ${G_c}=\left\{ {A,~B,~C,F} \right\}$. So, the dataset ${X_G}$ is represented by ${X_{{G_c}}}$ in new feature space.

$${X_{{G_c}}}=\left[ {\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} 1&3 \end{array}}&4&8 \\ {\begin{array}{*{20}{c}} 3&6 \end{array}}&5&1 \\ {\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} 5&9 \end{array}} \\ {\begin{array}{*{20}{c}} 2&5 \end{array}} \end{array}}&{\begin{array}{*{20}{c}} 4 \\ 4 \end{array}}&{\begin{array}{*{20}{c}} 6 \\ 2 \end{array}} \end{array}} \right]$$

Employing ${X_{{G_c}}}$, the next steps of SPC-BAHSIC is the same as SP-BAHSIC. Again, using RBF kernel function as $\phi (.)$, the samples similarity matrix $K_{{\{ A,B,C,F\} }}^{\phi }$ is computed.

$$K_{{\{ A,B,C,F\} }}^{\phi }=\left[ {\begin{array}{*{20}{c}} 1&{0.28}&{\begin{array}{*{20}{c}} {0.33}&{0.44} \end{array}} \\ {0.28}&1&{\begin{array}{*{20}{c}} {0.46}&{0.92} \end{array}} \\ {\begin{array}{*{20}{c}} {0.33} \\ {0.44} \end{array}}&{\begin{array}{*{20}{c}} {0.46} \\ {0.92} \end{array}}&{\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} 1 \\ {0.44} \end{array}}&{\begin{array}{*{20}{c}} {0.44} \\ 1 \end{array}} \end{array}} \end{array}} \right]$$

Also, $\psi (.)$ is used to compute the following similarity matrices:

$$K_{{\{ B,C,F\} }}^{\psi }=\left[ {\begin{array}{*{20}{c}} 1&{0.74}&{\begin{array}{*{20}{c}} {0.82}&{0.82} \end{array}} \\ {0.74}&1&{\begin{array}{*{20}{c}} {0.84}&{0.98} \end{array}} \\ {\begin{array}{*{20}{c}} {0.82} \\ {0.82} \end{array}}&{\begin{array}{*{20}{c}} {0.84} \\ {0.98} \end{array}}&{\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} 1 \\ {0.85} \end{array}}&{\begin{array}{*{20}{c}} {0.85} \\ 1 \end{array}} \end{array}} \end{array}} \right],$$

$$K_{{\{ A,C,F\} }}^{\psi }=\left[ {\begin{array}{*{20}{c}} 1&{0.76}&{\begin{array}{*{20}{c}} {0.90}&{0.83} \end{array}} \\ {0.76}&1&{\begin{array}{*{20}{c}} {0.86}&{0.98} \end{array}} \\ {\begin{array}{*{20}{c}} {0.90} \\ {0.83} \end{array}}&{\begin{array}{*{20}{c}} {0.86} \\ {0.98} \end{array}}&{\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} 1 \\ {0.88} \end{array}}&{\begin{array}{*{20}{c}} {0.88} \\ 1 \end{array}} \end{array}} \end{array}} \right],$$

$$K_{{\{ A,B,F\} }}^{\psi }=\left[ {\begin{array}{*{20}{c}} 1&{0.73}&{\begin{array}{*{20}{c}} {0.75}&{~~~0.81} \end{array}} \\ {0.73}&1&{\begin{array}{*{20}{c}} {0.83}&{~~~0.98} \end{array}} \\ {\begin{array}{*{20}{c}} {0.75} \\ {0.81} \end{array}}&{\begin{array}{*{20}{c}} {0.83} \\ {0.98} \end{array}}&{\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} {1~~~} \\ {0.81~~~} \end{array}}&{\begin{array}{*{20}{c}} {0.81} \\ 1 \end{array}} \end{array}} \end{array}} \right],$$

$$K_{{\{ A,B,C\} }}^{\psi }=\left[ {\begin{array}{*{20}{c}} 1&{0.93}&{\begin{array}{*{20}{c}} {0.77}&{~~0.97} \end{array}} \\ {0.93}&1&{\begin{array}{*{20}{c}} {~~0.93~~}&{0.98~~} \end{array}} \\ {\begin{array}{*{20}{c}} {0.77} \\ {0.97} \end{array}}&{\begin{array}{*{20}{c}} {0.93} \\ {0.98} \end{array}}&{\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} {1~~~} \\ {0.88~~~} \end{array}}&{\begin{array}{*{20}{c}} {0.88~} \\ 1 \end{array}} \end{array}} \end{array}} \right]$$

Similar to the first example, HSIC₂ is computed between $K_{{\{ A,B,C,F\} }}^{\phi }$ and each of $K_{{\{ B,C,F\} }}^{\psi }$, $K_{{\{ A,C,F\} }}^{\psi }$, $K_{{\{ A,B,F\} }}^{\psi }$ and $K_{{\{ A,B,C\} }}^{\psi }$ and their values are stored in ${H_1}$.

$${H_1}=\left[ {\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} {0.0133}&{0.0111} \end{array}}&{\begin{array}{*{20}{c}} {0.0151}&{0.0064} \end{array}} \end{array}} \right]$$

From ${H_1}$, it is clear that the similarity measure between $K_{{\{ A,B,C,F\} }}^{\phi }$ and $K_{{\{ A,B,F\} }}^{\psi }$ is the highest and so, feature $C$ should be removed from ${G_c}$. In the second phase, these similarity matrices are achieved.

$$K_{{\{ B,F\} }}^{\psi }=\left[ {\begin{array}{*{20}{c}} 1&{0.75}&{\begin{array}{*{20}{c}} {0.82}&{0.82} \end{array}} \\ {0.75}&1&{\begin{array}{*{20}{c}} {0.84}&{0.99} \end{array}} \\ {\begin{array}{*{20}{c}} {0.82} \\ {0.82} \end{array}}&{\begin{array}{*{20}{c}} {0.84} \\ {0.99} \end{array}}&{\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} 1 \\ {0.85} \end{array}}&{\begin{array}{*{20}{c}} {0.85} \\ 1 \end{array}} \end{array}} \end{array}} \right],$$

$$K_{{\{ A,F\} }}^{\psi }=\left[ {\begin{array}{*{20}{c}} 1&{0.77}&{\begin{array}{*{20}{c}} {0.90}&{0.83} \end{array}} \\ {0.77}&1&{\begin{array}{*{20}{c}} {0.86}&{0.99} \end{array}} \\ {\begin{array}{*{20}{c}} {0.90} \\ {0.83} \end{array}}&{\begin{array}{*{20}{c}} {0.86} \\ {0.99} \end{array}}&{\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} 1 \\ {0.88} \end{array}}&{\begin{array}{*{20}{c}} {0.88} \\ 1 \end{array}} \end{array}} \end{array}} \right],$$

$$K_{{\{ A,B\} }}^{\psi }=\left[ {\begin{array}{*{20}{c}} 1&{0.94}&{\begin{array}{*{20}{c}} {0.77}&{0.97} \end{array}} \\ {0.94}&1&{\begin{array}{*{20}{c}} {0.94}&{0.99} \end{array}} \\ {\begin{array}{*{20}{c}} {0.77} \\ {0.97} \end{array}}&{\begin{array}{*{20}{c}} {0.94} \\ {0.99} \end{array}}&{\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} 1 \\ {0.88} \end{array}}&{\begin{array}{*{20}{c}} {0.88} \\ 1 \end{array}} \end{array}} \end{array}} \right],$$

Using (14), HSIC₂ is computed between $K_{{\{ A,B,C,F\} }}^{\phi }$ and each of $K_{{\{ B,F\} }}^{\psi }$, $K_{{\{ A,F\} }}^{\psi }$ and $K_{{\{ A,B\} }}^{\psi }$, as ${H_2}$ represents:

$${H_2}=\left[ {\begin{array}{*{20}{c}} {0.0132}&{0.0110}&{0.0063} \end{array}} \right]$$

According to ${H_2}$, feature $A$ is the next eliminated feature. So, features $B$ and $F$ are the most informative features which are returned by algorithm of SPC-BAHSIC.

In this part, we run SP-FOHSIC on dataset ${X_G}$ with $m=4$ samples and $n=4$ features, named $G=\left\{ {A,~B,~C,D} \right\}$ and $n'=2$ of informative features are needed.

$${X_G}=\left[ {\begin{array}{*{20}{c}} 1&3&{\begin{array}{*{20}{c}} 4&2 \end{array}} \\ 3&6&{\begin{array}{*{20}{c}} 5&4 \end{array}} \\ {\begin{array}{*{20}{c}} 5 \\ 2 \end{array}}&{\begin{array}{*{20}{c}} {10} \\ 5 \end{array}}&{\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} 4&3 \end{array}} \\ {\begin{array}{*{20}{c}} 4&4 \end{array}} \end{array}} \end{array}} \right]$$

By using RBF kernel function as $\phi (.)$, the samples similarity matrix $K_{{\{ A,B,C,D\} }}^{\phi }$ is computed as:

$$K_{{\{ A,B,C,D\} }}^{\phi }=\left[ {\begin{array}{*{20}{c}} 1&{0.70}&{\begin{array}{*{20}{c}} {0.27}&{0.83} \end{array}} \\ {0.70}&1&{\begin{array}{*{20}{c}} {0.64}&{0.94} \end{array}} \\ {\begin{array}{*{20}{c}} {0.27} \\ {0.83} \end{array}}&{\begin{array}{*{20}{c}} {0.64} \\ {0.94} \end{array}}&{\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} 1 \\ {0.50} \end{array}}&{\begin{array}{*{20}{c}} {0.50} \\ 1 \end{array}} \end{array}} \end{array}} \right]$$

Similarly, using RBF kernel for $\psi (.)$, the samples similarity matrices $K_{{\left\{ A \right\}}}^{\psi }$ (samples are only included feature A), $K_{{\{ B\} }}^{\psi }$ (samples are only included feature B), $K_{{\{ C\} }}^{\psi }$ (samples are only included feature B) and $K_{{\{ D\} }}^{\psi }$ (samples are only included feature D) are obtained:

$$K_{{\{ A\} }}^{\psi }=\left[ {\begin{array}{*{20}{l}} 1&{0.98}&{0.92}&{0.99} \\ {0.98}&1&{0.98}&{0.99} \\ {0.92}&{0.98}&1&{0.96} \\ {0.99}&{0.99}&{0.96}&1 \end{array}} \right],$$

$$K_{{\{ B\} }}^{\psi }=\left[ {\begin{array}{*{20}{l}} 1&{0.96}&{0.78}&{0.98} \\ {0.96}&1&{0.92}&{0.99} \\ {0.78}&{0.92}&1&{0.88} \\ {0.98}&{0.99}&{0.88}&1 \end{array}} \right],$$

$$K_{{\{ C\} }}^{\psi }=\left[ {\begin{array}{*{20}{l}} 1&{0.99}&1&1 \\ {0.99}&1&{0.99}&{0.99} \\ 1&{0.99}&1&1 \\ 1&{0.99}&1&1 \end{array}} \right],$$

$$K_{{\{ D\} }}^{\psi }=\left[ {\begin{array}{*{20}{l}} 1&{0.98}&{0.99}&{0.98} \\ {0.98}&1&{0.99}&1 \\ {0.99}&{0.99}&1&{0.99} \\ {0.98}&1&{0.99}&1 \end{array}} \right]$$

Using (14), HSIC₂ is computed between $K_{{\{ A,B,C,D\} }}^{\phi }$ and each of $K_{{\{ A\} }}^{\psi }$, $K_{{\{ B\} }}^{\psi }$, $K_{{\{ C\} }}^{\psi }$ and $K_{{\{ D\} }}^{\psi }$. ${H_1}$ is achieved as:

$${H_1}=\left[ {\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} {0.0025}&{0.0071} \end{array}}&{\begin{array}{*{20}{c}} {2.1554 \times {{10}^{ - 5}}}&{1.7748 \times {{10}^{ - 4}}} \end{array}} \end{array}} \right]$$

According to ${H_1}$, the HSIC₂ measure between $K_{{\{ A,B,C,D\} }}^{\phi }$ and $K_{{\{ B\} }}^{\psi }$ is maximum. So, among 4 features, feature $B$ is more suitable for selection. In the second phase, the samples similarity matrices $K_{{\{ {\text{B}},A\} }}^{\psi }$, $K_{{\{ B,C\} }}^{\psi }$ and $K_{{\{ {\text{B}},D\} }}^{\psi }$ are computed.

$$K_{{\{ B,A\} }}^{\psi }=\left[ {\begin{array}{*{20}{l}} 1&{0.94}&{0.72}&{0.97} \\ {0.94}&1&{0.90}&{0.99} \\ {0.72}&{0.90}&1&{0.84} \\ {0.97}&{0.99}&{0.84}&1 \end{array}} \right],$$

$$K_{{\{ B,C\} }}^{\psi }=\left[ {\begin{array}{*{20}{l}} 1&{0.95}&{0.78}&{0.98} \\ {0.95}&1&{0.92}&{0.99} \\ {0.78}&{0.92}&1&{0.88} \\ {0.98}&{0.99}&{0.88}&1 \end{array}} \right],$$

$$K_{{\{ B,D\} }}^{\psi }=\left[ {\begin{array}{*{20}{l}} 1&{0.94}&{0.78}&{0.96} \\ {0.94}&1&{0.92}&{0.99} \\ {0.78}&{0.92}&1&{0.88} \\ {0.96}&{0.99}&{0.88}&1 \end{array}} \right]$$

Also, HSIC₂ value between $K_{{\{ A,B,C,D\} }}^{\phi }$ and each of them is stored in ${H_2}$ as:

$${H_2}=\left[ {\begin{array}{*{20}{c}} {0.0093}&{0.0071}&{0.0072} \end{array}} \right]$$

From HSIC₂ values in ${H_2}$, the next candidate for selection is feature $A$. Thus, features $A$ and $B$ are returned by SP-FOHSIC as the most informative features.

As the last example, SPC-FOHSIC is run on dataset ${X_G}$ with $m=4$ samples and $n=6$ features, named $G=\left\{ {A,~B,~C,D,E,F} \right\}$. As in previous experiments, our aim is to find $n'=2$ most informative features.

$${X_G}=\left[ {\begin{array}{*{20}{l}} 1&3&4&2&4&8 \\ 3&6&5&4&2&1 \\ 5&9&4&3&7&6 \\ 2&5&4&4&3&2 \end{array}} \right]$$

Again, by using gap statistic method, these 6 features can be grouped into $l=4$ clusters. These clusters are determined via k-means method. Using their centroids, the features ${G_c}=\left\{ {A,~B,~C,F} \right\}$ establish the new feature space wherein the dataset ${X_G}$ is represented as ${X_{{G_c}}}$:

$${X_{{G_c}}}=\left[ {\begin{array}{*{20}{l}} 1&3&4&8 \\ 3&6&5&1 \\ 5&9&4&6 \\ 2&5&4&2 \end{array}} \right]$$

The next steps of SPC-FOHSIC is the same as SP-FOHSIC. Again, using RBF kernel function as $\phi (.)$, the samples similarity matrix $K_{{\{ A,B,C,F\} }}^{\phi }$ is computed as:

$$K_{{\{ A,B,C,F\} }}^{\phi }=\left[ {\begin{array}{*{20}{l}} 1&{0.28}&{0.33}&{0.44} \\ {0.28}&1&{0.46}&{0.92} \\ {0.33}&{0.46}&1&{0.44} \\ {0.44}&{0.92}&{0.44}&1 \end{array}} \right]$$

Also, $\psi (.)$ is used to compute the following similarity matrices:

$$K_{{\{ A\} }}^{\psi }=\left[ {\begin{array}{*{20}{l}} 1&{0.98}&{092}&{0.99} \\ {0.98}&1&{0.98}&{0.99} \\ {0.92}&{0.98}&1&{0.95} \\ {0.99}&{0.99}&{0.96}&1 \end{array}} \right],$$

$$K_{{\{ B\} }}^{\psi }=\left[ {\begin{array}{*{20}{l}} 1&{0.96}&{0.83}&{0.98} \\ {0.96}&1&{0.96}&{0.99} \\ {0.83}&{0.96}&1&{0.92} \\ {0.98}&{0.99}&{0.92}&1 \end{array}} \right],$$

$$K_{{\{ C\} }}^{\psi }=\left[ {\begin{array}{*{20}{l}} 1&{0.99}&1&1 \\ {0.99}&1&{0.99}&{0.99} \\ 1&{0.99}&1&1 \\ 1&{0.99}&1&1 \end{array}} \right],$$

$$K_{{\{ F\} }}^{\psi }=\left[ {\begin{array}{*{20}{l}} 1&{0.78}&{0.98}&{~~0.83} \\ {0.78}&1&{0.88}&{0.99~~} \\ {0.98}&{0.88}&1&{0.92~} \\ {0.83}&{0.99}&{0.92}&1 \end{array}} \right]$$

Similarly, HSIC₂ is computed between $K_{{\{ A,B,C,F\} }}^{\phi }$ and each of $K_{{\{ A\} }}^{\psi }$, $K_{{\{ B\} }}^{\psi }$, $K_{{\{ C\} }}^{\psi }$ and $K_{{\{ F\} }}^{\psi }$ and their values are stored in ${H_1}$ as:

$${H_1}=\left[ {\begin{array}{*{20}{c}} {\begin{array}{*{20}{c}} {0.0020}&{0.0044} \end{array}}&{\begin{array}{*{20}{c}} {9.6213 \times {{10}^{ - 5}}}&{0.0091} \end{array}} \end{array}} \right]$$

From ${H_1}$, it is clear that the similarity measure between $K_{{\{ A,B,C,F\} }}^{\phi }$ and $K_{{\{ F\} }}^{\psi }$ is the highest and so, feature $F$ should be selected. In the second phase, these similarity matrices are achieved as:

$$K_{{\{ F,A\} }}^{\psi }=\left[ {\begin{array}{*{20}{l}} 1&{0.77}&{0.90}&{0.83} \\ {0.77}&1&{0.86}&{0.99} \\ {0.90}&{0.86}&1&{0.88} \\ {0.83}&{0.99}&{0.88}&1 \end{array}} \right],$$

$$K_{{\{ F,B\} }}^{\psi }=\left[ {\begin{array}{*{20}{l}} 1&{0.75}&{0.82}&{0.82} \\ {0.75}&1&{0.84}&{0.99} \\ {0.82}&{0.84}&1&{0.85} \\ {0.82}&{0.99}&{0.85}&1 \end{array}} \right]$$

$$K_{{\{ F,C\} }}^{\psi }=\left[ {\begin{array}{*{20}{l}} 1&{0.78}&{0.98}&{0.83} \\ {0.78}&1&{0.88}&{0.99} \\ {0.98}&{0.88}&1&{0.92} \\ 1&{0.92}&{0.99}&{0.83} \end{array}} \right]$$

Using (14), HSIC₂ is computed between $K_{{\{ A,B,C,F\} }}^{\phi }$ and each of $K_{{\{ F,A\} }}^{\psi }$, $K_{{\{ F,B\} }}^{\psi }$ and $K_{{\{ F,C\} }}^{\psi }$, represents as ${H_2}$:

$${H_2}=\left[ {\begin{array}{*{20}{c}} {0.0110}&{0.0132}&{0.0092} \end{array}} \right]$$

According to ${H_2}$, feature $B$ is the next informative feature. So, features $B$ and $F$ are the most informative features which are returned by algorithm of SPC-FOHSIC.