A Multi-objective hybrid filter-wrapper evolutionary approach for feature selection

Abstract

Feature selection is an important pre-processing data mining task, which can reduce the data dimensionality and improve not only the classification accuracy but also the classifier efficiency. Filters use statistical characteristics of the data as the evaluation measure rather than using a classification algorithm. On the contrary, the wrapper process is computationally expensive because the evaluation of every feature subset requires running the classifier on the datasets and computing the accuracy from the obtained confusion matrix. In order to solve this problem, we propose a hybrid tri-objective evolutionary algorithm that optimizes two filter objectives, namely the number of features and the mutual information, and one wrapper objective corresponding to the accuracy. Once the population is classified into different non-dominated fronts, only feature subsets belonging to the first (best) one are improved using the indicator-based multi-objective local search. Our proposed hybrid algorithm, named Filter-Wrapper-based Nondominated Sorting Genetic Algorithm-II, is compared against several multi-objective and single-objective feature selection algorithms on eighteen benchmark datasets having different dimensionalities. Experimental results show that our proposed algorithm gives competitive and better results with respect to existing algorithms.

Appendix A

1.1 A.1 FW-NSGA-II results using C4.5

Table 7 shows that the number of features selected by FW-NSGA-II is smaller than the total number of features when using C4.5 in the wrapper evaluations. It also increases the classification accuracy on almost all low- and medium-dimensional datasets. Compared with WrapperAlgo, FW-NSGA-II outperformed the WrapperAlgo in terms of accuracy on almost all datasets. Comparing FW-NSGA-II with FilterAlgo, FastAlgo, and RapidPSO, FW-NSGA-II outperformed FilterAlgo, FastPSO, and RapidPSO in terms of the classification performance and the size of feature subset in most cases in low- and medium-dimensional datasets. FW-NSGA-II tends to produce larger feature subsets than WrapperAlgo, GRASP and Random Forest, but smaller subsets than FastPSO, RapidPSO, and FilterAlgo. FW-NSGA-II outperformed the GRASP and the RF algorithm in terms of accuracy in almost all low- and medium-dimensional datasets.

For high-dimensional datasets, FW-NSGA-II outperformed FilterAlgo, FastAlgo, RapidPSO and GRASP in terms of the classification performance and the number of features. However, the Random Forest algorithm achieved better performance than FW-NSGA-II and all the other competitive algorithms in terms of the number of features and the classification accuracy in almost all datasets. The number of features selected by FW-NSGA-II in high-dimensional datasets is higher than the number of features selected by FW-NSGA-II in low- and medium-dimensional datasets. In fact, the number of removed features in FW-NSGA-II for the Ovarian dataset is around 20%, which is different from the number of removed features in low- and medium-dimensional datasets which is around 50% of the original feature size.

The Decision Tree classifier improved the performance of FW-NSGA-II in terms of the classification accuracy and the number of features, compared with the K-NN classifier. The results show that FW-NSGA-II achieved better results than the other algorithms in almost all low- and medium-dimensional datasets, but this is not the case with high-dimensional datasets. In fact, the Random Forest algorithm outperformed the FW-NSGA-II due to its ability to handle large datasets with higher dimensionality compared with the Decision Tree classifier. The source code and the experimental data could be obtained upon request from the corresponding author.