A multi-objective micro genetic ELM algorithm

Abstract

The extreme learning machine (ELM) is a methodology for learning single-hidden layer feedforward neural networks (SLFN) which has been proved to be extremely fast and to provide very good generalization performance. ELM works by randomly choosing the weights and biases of the hidden nodes and then analytically obtaining the output weights and biases for a SLFN with the number of hidden nodes previously fixed. In this work, we develop a multi-objective micro genetic ELM $(\mathrm{\mu }\mathrm{G}\mathrm{-}\mathrm{ELM})$ which provides the appropriate number of hidden nodes for the problem being solved as well as the weights and biases which minimize the MSE. The multi-objective algorithm is conducted by two criteria: the number of hidden nodes and the mean square error (MSE). Furthermore, as a novelty, $\mathrm{\mu }\mathrm{G}\mathrm{-}\mathrm{ELM}$ incorporates a regression device in order to decide whether the number of hidden nodes of the individuals of the population should be increased or decreased or unchanged. In general, the proposed algorithm reaches better errors by also implying a smaller number of hidden nodes for the data sets and competitors considered.

Introduction

Over the last twenty years artificial neural networks (ANN) have allowed researchers to model and solve real problems that cannot be easily solved with classical mathematical tools (due to a great number of data and/or variables, non-linear relations, etc.).

Historically back-propagation algorithms (BP) are the most widely used for training SLFNs. In order to set all their weights and biases, from input to hidden nodes and from hidden to output nodes, they require many parameters, they are slow and the training phase has to be repeated frequently.

However, the ELM does not need to adjust, in a computational expensive way, the hidden weights and biases since it chooses them at random. Then, with these weights and biases the first part of the SLFN is evaluated. Due to the fact that the activation function in the output layer is linear, the resulting values allow us to define a linear system of equations whose solutions are the output weights and biases of the SLFN. This linear system is easily solved through a simple generalized inverse calculation. To justify the correctness of the ELM algorithm it has been proved in [1] that, under certain regularity conditions, given a set of patterns with N records and any small positive value $ϵ>0$, there exists a number of hidden nodes $\tilde{N}$, $\tilde{N}\le N$, such that, after applying the above process the learning approximation error of the SLFN is less than $ϵ$ with probability one. Since the publication of the original ELM a great number of different alternatives have appeared: I-ELM [1], OS-ELM [2], EI-ELM [3], OP-ELM [4], EM-ELM [5], EOS-ELM [6], CEOS-ELM [7], TS-ELM [8], TROP-ELM [9] and EELM [10], some of which we will use in order to carry out our comparisons in Section 5.

On the other hand, evolutionary algorithms (EAs) are search and optimization methods based on the principles of natural evolution and the genetics that try to approximate the optimal solution of a given problem [11]. The general schema of an evolutionary algorithm is shown in Procedure 1. Its main sub-processes include the variation, the evaluation and the selection of solutions. The EAs in general, and multi-objective EAs [12], in particular, have shown their great potential during the last two decades. A list of references with more than 6500 on evolutionary multi-objective optimization is maintained by Coello [13].

Procedure 1

General schema of an evolutionary algorithm.

In this paper we develop an improved version of our micro genetic algorithm, $\mathrm{\mu }\mathrm{G}\mathrm{-}\mathrm{ELM}$. We introduced the original version in [14], where our main contribution was the development of a bi-objective genetic algorithm which determined simultaneously the appropriate number of hidden nodes of the artificial neural network as well as the corresponding set of weights and biases, unlike other authors did, for instance [15] and [16], which only use the approximation error to guide the search of the appropriate weights and biases given a fixed number of hidden nodes. In order to conduct the search of the appropriate number of hidden nodes we proposed a new strategy based on regression by means of which the algorithm decides, in each iteration, to increase, to decrease or to unchange the number of hidden nodes of the artificial neural networks in the current population being evolved. Furthermore, we developed a new mutation operator for the number of hidden nodes and a survival selection operator, both specially designed for our algorithm.

The main goal of this paper is to look further in depth at the development of the $\mathrm{\mu }\mathrm{G}\mathrm{-}\mathrm{ELM}$ algorithm. Since the regression strategy uses the information of certain solutions and, also, this information can be used with or without smoothing, in this paper we propose and study new alternatives for the original regression device resulting from the combination of three ways of selecting the solutions and three ways of preprocessing their information. We determine the best one and the selected alternative is then compared with other usual ELM algorithms in the literature. Furthermore, several elements of the original algorithm have been improved. The regression device considers more types of relations (Section 4) and the tournament selection process used to build the micro population to be evolved depends on the result of the regression device now (Section 3.4).

The paper is organized as follows: In Section 2 we show the previous establishments and notations for working with SLFNs and we introduce the bi-objective optimization problem whose solutions we approximate. Section 3 presents the $\mathrm{\mu }\mathrm{G}\mathrm{-}\mathrm{ELM}$ algorithm with a detailed description of its steps. Section 4 is devoted to describing the regression device under several alternatives. Section 5 shows two experiments. The first one corresponds to a detailed study of the performance of our algorithm under several choices. In the second, we compare our $\mathrm{\mu }\mathrm{G}\mathrm{-}\mathrm{ELM}$ under the best implementation selected as a result of the first experiment with the chosen competitors. Summary and concluding remarks are provided in Section 6.