Feature selection based on rough set approach, wrapper approach, and binary whale optimization algorithm

Abstract

The principle of any approach for solving feature selection problem is to find a subset of the original features. Since finding a minimal subset of the features is an NP-hard problem, it is necessary to develop and propose practical and efficient heuristic algorithms. The whale optimization algorithm is a recently developed nature-inspired meta-heuristic optimization algorithm that imitates the hunting behavior of humpback whales to solve continuous optimization problems. In this paper, we propose a novel binary whale optimization algorithm (BWOA) to solve feature selection problem. BWOA is especially desirable and appealing for feature selection problem whenever there is no heuristic information that can lead the search to the optimal minimal subset. Nonetheless, whales can find the best features as they hunt the prey. Rough set theory (RST) is one of the effective algorithms for feature selection. We use RST with BWOA as the first experiment, and in the second experiment, we use a wrapper approach with BWOA on three different classifiers for feature selection. Also, we verify the performance and the effectiveness of the proposed algorithm by performing our experiments using 32 datasets from the UCI machine learning repository and comparing the proposed algorithm with some powerful existing algorithms in the literature. Furthermore, we employ two nonparametric statistical tests, Wilcoxon Signed-Rank test, and Friedman test, at 5% significance level. Our results show that the proposed algorithm can provide an efficient tool to find a minimal subset of the features.

Appendix

1. Logistic regression (LR) [48, 49, 49]  A common method that is used for classification and known as the exponential or log-linear classifiers. LR is a selective learning classifier that directly estimates the parameters of the posterior distribution function P(c|x). This algorithm assumes the distribution P(c|x) is given by Eq. (22),

$$\begin{aligned} P(c=k|x) = \frac{\exp \left( w_k^Tx\right) }{\sum \nolimits _{j=1}^{K}\exp \left( w_j^Tx\right) } \end{aligned}$$

(22)

where $w_js$ are the parameters to estimate and K is the number of classes. Then maximum likelihood method is used to approximate $w_js$. Since the Hessian matrix for the logistic regression model is positive definite, the error function has a unique minimum. In this proposed system, the LR is used as a classification to ensure the goodness of the selected features. The best feature combination is the one with maximum classification performance and minimum number of selected features. Note that we use the package of Pattern Recognition and Machine Learning Toolbox (PRML) in Matlab which provides logistic regression functions for both binary and multiclass classification problems [81].

2. C4.5 decision tree classifier [50, 51]  The C4.5 technique is one of the decision tree families that can produce both decision tree and rule-sets, and construct a tree to improve prediction accuracy.

C4.5 uses two heuristic criteria to rank possible tests  Information gain that uses attribute selection measure, which minimizes the total entropy of the subset ${S_i}$, and the default gain ratio that divides information gain by the information provided by the test outcomes. The information gain algorithm is described as the function gain (A), which is shown below:

• Select the attribute with the highest information gain.

• S contains $s_i$ tuples of class $C_i$ for $i = {1,\ldots , m}$.

• Information measure or expected information is required to classify any arbitrary tuple:

  $$\begin{aligned} I(S_1, S_2,\ldots ,S_m)=-\sum _{i=1}^{m}\frac{S_i}{S}\log _2\frac{S_i}{S}. \end{aligned}$$

  (23)

• Entropy of attribute A with values ${a_1,a_2,\ldots ,a_v}$:

  $$\begin{aligned} E(A)=\sum _{j=1}^{v}\frac{S_{1j}+\cdots +S_{mj}}{S} I(S_{1j},\ldots ,S_{mj}). \end{aligned}$$

  (24)

• Information gain means how much can be gained by branching on attribute A:

  $$\begin{aligned} Gain(A)=I(S_1, S_2,\ldots ,S_m)-E(A). \end{aligned}$$

  (25)

3. Naïve Bayes (NB) [52, 53]  Naïve Bayes has proven to be a simple, useful, and powerful machine learning approach in classification studies. NB is recognized as a simple Bayesian classification algorithm. NB classifier is highly scalable, requiring a number of parameters linear in the number of variables (features/predictors) in a learning problem. Maximum-likelihood training can be employed y calculating a closed-form expression, which takes linear time, rather than by expensive iterative approximation as used for many other types of classifiers.