Evolutionary parallel extreme learning machines for the data classification problem

Highlights

• First evolutionary parallel ELM algorithm for the data classification problem.

• The ELM is enhanced with the feature selection.

• The proposed algorithm tunes its parameters at run time.

• The scalability of the proposed algorithm is (near)-linear.

• The state-of-the-art algorithms are outperformed.

Abstract

This study proposes an Island Parallel Evolutionary Extreme Learning Machine algorithm (IPE-ELM) for the well-known data classification problem. The ELM is a fast and efficient machine learning technique with its single-hidden layer feed-forward neural network (SLFN). High prediction accuracy and learning speed of the ELM make it an elegant tool for the fitness calculation process of the evolutionary algorithms. The IPE-ELM algorithm combines the evolutionary genetic algorithms (for feature selection), ELM machine learning technique (for prediction accuracy calculation), parallel computation (for faster fitness evaluation), and parameter tuning (activation function selection and the number of hidden neurons) for the solution of this important problem. Each ELM that runs at a different processor selects one of four different activation functions (Sine, Cosine, Sigmoid and Hyperbolic Tangent) and uses a randomized number of hidden neurons to achieve higher prediction accuracy. The proposed algorithm provides high quality results with its (near)-linear scalability behavior. The IPE-ELM algorithm is compared with state-of-the-art data classification algorithms by using UCI benchmark datasets and significant improvements are reported in terms of prediction accuracy with reasonable execution times. The scalable IPE-ELM algorithm can be reported as the first island parallel evolutionary classification algorithm with its high prediction accuracy results that outperforms state-of-the-art algorithms in literature.

Introduction

Data classification is a crucial mining technique with its many applications in our daily life (Ian, Witten, & Frank, 2011). Scientists can identify, acquire knowledge and derive statistical/predictive models by making use of data classification techniques. The accuracy and the execution speed are important issues of the data classification process. One of the best means of extracting interesting and valuable patterns is making use of recent machine learning techniques. Dealing with large datasets (having many features and rows) requires advanced supervised machine learning techniques (Nasrabadi, 2007, Unler and Murat, 2010). Supervised machine learning techniques build a model to predict the class labels of data by using a set of training data. Recently, it is very common to have a large amount of data with several attributes/features and it is not an easy process to properly clean and classify such a large amount of data correctly and obtain the distilled information. In literature, there have been many supervised techniques proposed for the solution of the data classification problem. However, the demand for fast algorithms that work with high prediction accuracy is still a valuable research area.

A high quality classifier is a crucial part of a data classification process. The classifier should have a good prediction accuracy and a good generalization ability. The training speed of a classifier is another important point that should be considered. Extreme Learning Machine (ELM) is a recent and fast supervised machine learning technique with its high performance for the data classification problem (Guang-Bin Huang & Zhu, 2004). The learning speed of feed-forward neural networks is generally slow and it has been a major drawback in machine learning applications. Slow gradient-based learning algorithms are used to train neural networks and the parameters of the networks are tuned by using such learning techniques. These are the main reasons of the slow learning process. However, the ELM is a different machine learning technique for single-hidden layer feed-forward neural networks (SLFNs) that randomly chooses the number of hidden nodes and determines the output weights of SLFNs (Guang-Bin Huang & Siew, 2006). This property of the ELM makes it a suitable technique for intensive fitness chromosome evaluation of evolutionary genetic algorithms that select the best feature subset. The ELM has been applied to a lot of important problems and many studies are still under progress for improving the performance of this valuable machine learning technique (Huang, Wang, & Lan, 2011).

With recent developments in computer science, the need for real-time processing of large datasets presents big challenges to traditional ways of data processing (Bolón-Canedo & Alonso-Betanzos, 2012). For this reason, Feature Subset Selection (FSS) has attracted the attention of scientists to filter out unnecessary data and greatly reduce processing time (Mingkui Tan & Tsang, 2014). Ensemble-based wrapper methods (they use an exploration method for efficient FSS and use a machine learning method to measure the accuracy level) applied with FSS are providing good results for the data classification problem (Xiaowei Xue & Wu, 2017). The wrapper methods are computationally expensive tools since they need to compute many fitness values for the explored subsets. However, they are still the best performing methods.

We use a parallel computation environment to select the features of a dataset by using a genetic algorithm and apply ELM to the selected features to evaluate the prediction accuracy through fast evaluation of the instances. Genetic and ELM machine learning have been used before for solving the data classification problem. However, it is the first application of these methods with a parallel island genetic algorithm approach. We tune the parameters of the ELM dynamically and experimentally show that our method outperforms state-of-the-art metaheuristics. There have been earlier works that try to parallelize the ELM. Our method differs from the fact that we don’t parallelize the matrix multiplication phase of ELM, which is a trivial method that enables faster execution using parallel processing. This process requires intensive communication between processors and not scalable. Instead, we propose an island parallel method and execute as many ELM as the number of processors in the environment simultaneously. This approach provides an effective diversification mechanism for improving the population quality of the genetic algorithm.

Considering all the issues mentioned above, we propose a novel evolutionary island parallel ELM-based classifier for the data classification problem. To the best of our knowledge, the IPE-ELM algorithm is the first island parallel machine learning algorithm in literature that has been applied to the data classification problem (Aggarwal, 2014). The IPE-ELM generates diversified populations at each processor’s memory and improves the population’s fitness qualities independently. Our approach provides a very effective diversification mechanism for increasing the population quality of the genetic algorithm by initializing random number generator of each processor with a different seed. At the termination phase of the processes at each processor, the best results of the slave nodes are collected by the master node and the overall best solution is reported.

Parallel machine learning is a new developing research area and designing scalable parallel algorithms in this field is challenging. In our opinion, the IPE-ELM algorithm is a unique algorithm with its efficient features when compared with other algorithms in this domain. Four different activation functions are used during the classification process, namely, Sine, Cosine, Sigmoid and Hyperbolic Tangent. It is not always possible to choose a single type of activation function to be the best one for every possible dataset. Different activation functions can do better on varied datasets. Each processor randomly selects one of these activation functions and continues its optimization process. The number of hidden neurons is another criterion to be considered during the classification. The tuning of this parameter greatly affects performance. This issue is also observed and the results obtained in our experiments are reported. Therefore, each processor decides a different number of hidden neurons (in the range of [2–10] % of the instance size) and runs the ELM. The size of the population has been chosen as 70 after comprehensive tests. The best performing convergence, truncation, crossover and mutation ratios are applied in all processors of the parallel distributed memory environment by using Message Passing Interface (MPI) libraries. The parallel and diversified populations of the IPE-ELM algorithm provide a stagnation prevention mechanism. At each processor, we select different seeds for the randomization of all parameters in the FSS and the ELM phases of the algorithm, which prevents the genetic algorithm from exploring the same areas of the search space repeatedly. Comprehensive experiments comparing our scalable algorithm with state-of-the-art classification algorithms show that the IPE-ELM algorithm outperforms them in terms of prediction accuracy values with reasonable execution times.

Some of the state-of-the-art metaheuristics that have been applied to the data classification problem are Particle Swarm Optimization (PSO) (Unler & Murat, 2010), Attribute Bagging (AB), (a technique for improving the accuracy and stability of classifier ensembles induced using random subsets of features) (Bryll & Gutierrez, 2003), Multi-View Adaboost (MVA) (Xu, 2010), Random Subspace Method for constructing decision forests (RSE) (Ho, 1998), Correlation based Feature Selection (CFS-SFS) (Hall, 1998), C4.5 (Quinlan, 1996), Hybrid Genetic Algorithm and ELM-based feature selection algorithm (HGEFS) (Xiaowei Xue & Wu, 2017), Advanced Binary Ant Colony Optimization (ABACO) (Shima Kashef, 2015), and ACO-based feature selection algorithm (ACOFS) (Chen, Chen, & Chen, 2013). The algorithms mentioned here need to calculate the fitness of each next possible better solution, which is the most time-consuming part of these algorithms. In our experiments, we compare our solutions with the results of these algorithms.

In Section 2, related studies for the state-of-the-art ELM techniques and data classification algorithms are given. The details of the ELM are presented in Section 3. The proposed IPE-ELM algorithm is introduced in Section 4. The setup of the experimental environment, obtained results of the experiments, and comparison with state-of-the-art methods are reported in Section 5. Concluding remarks are provided in the last section.