Enhancing ELM by Markov Boundary based feature selection

Abstract

ELM, as an efficient classification technology, has been used in many popular application domains. However, ELM has weak generalization performance when the data set is small with respect to its feature space. In this paper, an enhanced ELM algorithm based on representative features is proposed to address the problem. At first, the method automatically generates some discrete intervals for every continuous feature. Then, it removes the irrelevant features by a method considering the feature interaction and reduces the weakly relevant features by a mutual information based method. Further, the reduction of redundancy features is conducted. Instead of constructing a large Bayesian network using all features, we just select the features of high relevance with the object node by an improved Markov Boundary identifying algorithm. Finally, we obtain the enhanced ELM classifier by training ELM using the extracted representative features and a genetic algorithm based weight assignment mechanism. The experiments conducted on real and synthetic small sample data sets demonstrate that the enhanced ELM classifier based on representative features outperforms the other methods used in our comparison study in terms of both efficiency and effectiveness.

Introduction

In recent years, classification problem has regained extensive research efforts from computer scientists, due to the explosive emergence of new classification applications, especially with the emergence of the big data. It is one of the challenges for the researcher on how to learn a model from the various data and classify the data quickly, such as protein sequences classification in bioinformatics, online social network prediction, XML document classification, mobile object data classification, cloud resource classification, online real-time stream data prediction and user-generated text documents from the Internet and so on [1], [2], [3], [4], [5]. How to classify the given data efficiently and effectively is an important problem.

Extreme learning machine (ELM) is becoming popular since it generally requires far less training time than the conventional learning machines [6], [7], [8], [9], [10], [11]. ELM has originally been developed based on Single-hidden Layer Feedforward Neural Networks (SLFN) in [12], [13], [14], where the hidden nodes parameters are chosen randomly. A Survey of ELM and its variants has been given in [1].

ELM has a better classification and prediction performance in many domains. However, it is of weak generalization performance when the original data set is small relative to its feature space. For example, in bioinformatics, a typical microarray data is often of severely limited number of samples, but of several orders of magnitude more features (genes) [15], [16]. According to the traditional learning theory, given n features, the required number of samples m for the reliable classifier learning should be on the scale of O(2ⁿ) [17]. However, even the minimum requirement (m = 10n) as a statistical “rule of thumb” is patently impractical for a real microarray dataset [17]. This poses a great challenge to ELM. In this case, selecting a small number of representative features showing distinct profiles in different classes of samples becomes highly necessary.

Feature subset selection is a promising way to deal with the data of small samples but of high dimensionality. Two basic approaches for feature selection have been proposed and studied for machine learning applications, the wrapper and the filter techniques. The wrapper methods use the predictive accuracy of a predetermined learning algorithm to determine the goodness of the selected subsets, and the accuracy of the learning algorithms is usually high. However, the generality of the selected features is limited and the computational complexity is high [18]. The filter methods are independent of learning algorithms, and thus with good generality. Their computational complexity is low, but the accuracy of the learning algorithms is not guaranteed [19], [20], [21]. The filter methods, due to their computational efficiency, are usually a good choice when the number of features is very large. However, the filter methods consider every attribute independently, ignoring the dependence relationship of attributes with each other.

Interestingly, Markov Boundary considers the dependency relationship between attributes [22]. Given a target variable, its Markov Boundary contains a minimal set of variables on which all other variables are conditionally independent of the target variable. Tsamardinos and Aliferis [23] provided the theoretical results that link the concepts of feature relevance in feature selection and Markov Boundary in Bayesian networks. The theoretical result proved that if a probability distribution can be faithfully represented by a Bayesian network, then the Markov Boundary of the class attribute in the Bayesian network is a unique and minimal feature subset for optimal feature selection. Further, some studies [24], [25] demonstrated that Markov Boundary feature selection outperforms most of the state-of-the-art feature selection algorithms. With these theoretical and practical results, the filter methods using Markov Boundary have attracted much attention in recent years [26], [27], [28]. In this paper, we develop a new Markov Boundary discovering algorithm and then compose the final Markov Boundary classifier in an efficient and effective way.

The main contributions of this paper are as follows: (1) We develop a method automatically determining the number of intervals for every feature of continuous value with theoretic guarantee. However, some commonly used methods often require an explicit parameter k to indicate the maximum number of discretized intervals; (2) We propose a method to identify representative feature subsets for each category. Instead of constructing a large Bayesian network using all features, we select those features of high relevance by just finding the Markov Boundary of the object node in an efficient way; (3) We devise an enhanced ELM for classification, namely MB-ELM. Unlike the existing methods, MB-ELM utilizes the selected feature subsets and a genetic algorithm based ensemble method to improve the effectiveness.

The remainder of this paper is organized as follows: Section 2 gives a brief overview of ELM. Section 3 presents the enhanced ELM classification framework, the four major elements of which, i.e. AIC¹-based discretization, initial filter, representative features selection and MB-ELM classifier construction, are detailed in Sections 3.1–3.4 respectively. In Section 4, we report extensive experimental results to validate the proposed MB-ELM. Finally, Section 5 conclude this paper.