Parsimonious regularized extreme learning machine based on orthogonal transformation

Abstract

Recently, two parsimonious algorithms were proposed to sparsify extreme learning machine (ELM), i.e., constructive parsimonious ELM (CP-ELM) and destructive parsimonious ELM (DP-ELM). In this paper, the ideas behind CP-ELM and DP-ELM are extended to the regularized ELM (RELM), thus obtaining CP-RELM and DP-RELM. For CP-RELM(DP-RELM), there are two schemes to realize it, viz. CP-RELM-I and CP-RELM-II(DP-RELM-I and DP-RELM-II). Generally speaking, CP-RELM-II(DP-RELM-II) outperforms CP-RELM-I(DP-RELM-I) in terms of parsimoniousness. Under nearly the same generalization, compared with CP-ELM(DP-ELM), CP-RELM-II(DP-RELM-II) usually needs fewer hidden nodes. In addition, different from CP-ELM and DP-ELM, for CP-RELM and DP-RELM the number of candidate hidden nodes may be larger than the number of training samples, which assists the selection of much better hidden nodes for constructing more compact networks. Finally, eleven benchmark data sets divided into two groups are utilized to do experiments and the usefulness of the proposed algorithms is reported.

Introduction

As a new learning algorithm for single-hidden-layer feed-forward neural networks (SLFN), the theory of extreme learning machines (ELMs) [1], [2], [3] has recently become increasingly popular in a wide scope of applications such as function approximation [4], [5], classification [5], [6], and density estimation [7]. The main reasons are due to their advantages of low computational cost, good generalization ability, and ease of implementation. Traditionally, all the parameters of the feed-forward networks are tuned with the commonly-used gradient descent-based methods. It is clear that gradient descent-based learning methods are generally very slow because of improper learning steps or easily stuck in the local minima. In contrast, the input weights and the hidden layer biases are chosen randomly in ELM, where the output weights can be analytically determined via the simple generalized inverse operation on the hidden layer output matrices. Hence, the learning speed of ELM is thousands of times faster than the traditional feed-forward network learning algorithms like back-propagation [8]. On the other side, ELM not only emphasizes to obtain the smallest training errors but also the smallest norm of weights. According to Bartlett׳s theory [9], the smaller the norm of weights is, the better generalization performance the networks tend to have. Therefore, ELM tends to have good generalization performance for feed-forward networks. Another well-known machine learning algorithm, viz. support vector machine (SVM) [10], [11], has been extensively used in classification and other fields in the last two decades due to its outstanding learning performance. In [12], ELM and SVM are compared from two viewpoints: one is the Vapnik–Chervonenkis dimension, and another is their performance under different training sample sizes. It is shown that the VC dimension of an ELM is equal to its number of hidden nodes with probability one. In addition, their generalization ability and computational complexity are demonstrated with changing training sample size. ELMs have weaker generalization ability than SVMs for small samples but can generalize as well as SVMs for large samples. Evidently, great superiority in computational speed especially for large-scale sample problems is found in ELMs.

In theory, ELM (with N hidden nodes) with randomly chosen input weights and hidden layer biases can exactly learn N distinct samples [13]. Actually, investigations show that N hidden nodes are usually not needed to construct good ELM. On one hand, the property of random choice of input weights and biases may generate redundant or unimportant hidden nodes for ELM, thus leading to the hidden layer output matrix being not of full column rank. In this context, ELM is ill-posed, often suffering from the deterioration of the generalization performance. On the other hand, according to the Occam׳s razor principle “plurality should not be posited without necessity” [14], it is known that a practical nonlinear modeling principle is to find the smallest model that generalizes well. Sparse¹ models are preferable in engineering applications since a model׳s computational complexity scales with its model complexity. Moreover, a sparse model is easier to interpret from the viewpoint of knowledge extraction [15]. Hence, it is necessary and important to find an sparse model for randomly generated hidden nodes, which owns good generalization performance meanwhile needing fewest hidden nodes. Additionally, many learning algorithms like online sequential learning [16], [17], [18] also need a thrifty mode before starts. That is, they were developed on the basis of ELM with an a prior fixed network size.

Generally speaking, two different approaches are usually utilized to sparsify ELM. The first is referred to as constructive algorithms, which begin with a small initial network (even a null model) and then gradually recruit the hidden nodes required until a satisfactory solution is sought. The incremental ELM [3] and its two variants [19], [20] belong to typical constructive algorithms. Subsequently, an error minimized extreme learning machine [21] was proposed, which can add random hidden nodes one by one or group by group and automatically determine the number of hidden nodes. Based on multi-response sparse regression and unbiased risk-estimation-based criterion C_p, two constructive algorithms [22], [23] for ELM were developed to cope with single- and multioutput regression problems, respectively. Recently, a constructive parsimonious extreme learning machine (CP-ELM) [24] was presented, which firstly transforms the hidden nodes output matrix into an upper triangular matrix equivalently using Givens rotation, and then recruits the hidden nodes incurring the largest additional residual error reductions to construct the final network subsequently. By contrast, the second, named as destructive algorithms (a.k.a. pruning algorithms), starts by training a larger than necessary network and then removes the redundant or unimportant hidden nodes. Aiming at classification problems, a pruned-ELM [25] was proposed to reach a compact network classifier, in which the irrelevant nodes are pruned by considering their relevance to the class labels from an initial large number of hidden nodes. In [26], an optimally pruned ELM (OP-ELM) methodology was presented, which firstly builds the original ELM, and then ranks the hidden nodes by applying multi-response sparse regression and selects the hidden nodes through leave-one-out validation. In parallel with CP-ELM, there is a destructive parsimonious ELM (DP-ELM) developed in [24], correspondingly.

To improve the performance on the original ELM, ELM was regularized using Tikhonov׳s method [27], i.e., regularized extreme learning machine (RELM) [28]. The sequential learning algorithm for RELM [29] was developed as well. And experimental studies [30], [31], [32], [33] demonstrated that RELM can overcome the ill-posed problem and control the machine learning complexity to enhance the generalization performance. Similar to the original ELM, the solution of RELM is also dense. Compared with a lot of efforts above-mentioned on sparsifying the solution of ELM, to date much less attention has been attracted to RELM. An improvement of OP-ELM was proposed to determine a compact network for RELM, which uses a cascade of two regularization penalties: first a L₁ penalty to rank the neurons of the hidden layer using least angle regression [34], followed by Tikhonov-regularized (TR) penalty on the regression weights for numerical stability and efficient pruning of the neurons, so this destructive algorithm was named TROP-ELM for short [35]. To obtain low complexity and sparse solution, a fast sparse approximation scheme for RELM (FSA-RELM) [36] was recently introduced, which begins with a null solution and gradually selects a new hidden node according to some criteria. And this procedure is repeated until the stopping criterion is met. Evidently, we know that FSA-RELM is a typical constructive algorithm.

Hence, we will do some research work to realize the sparseness of RELM, which includes the extensions of CP-ELM and DP-ELM to RELM. The main contributions in this paper are listed as:

• Enlightened by the ideas of CP-ELM and DP-ELM, we extend them to RELM, thus yielding CP-RELM and DP-RELM, respectively. CP-RELM and DP-RELM dominate CP-ELM and DP-ELM in terms of the number of hidden nodes, respectively, under nearly the same prediction accuracy.

• During the process of sparsifying RELM, two schemes are adopted. Although these two schemes are completely equivalent from theoretical viewpoint, their empirical studies show different to some extent.

• In CP-ELM and DP-ELM, the number of candidate hidden nodes is small because we need to transform the candidate hidden nodes output matrix into an upper triangular matrix by the orthogonal decomposition. If the number of candidate hidden nodes is large, the hidden nodes output matrix is singular or nearly singular. Due to the Tikhonov regularization, we can initialize RELM with more candidate hidden nodes (even their number is larger than the number of training samples) in order that much better hidden nodes are recruited to improve the generalization performance and generate more compact network.

The rest of the paper is organized as follows. In Section 2, ELM and RELM are introduced. And then the ideas behind CP-ELM and DP-ELM are described. In Section 4, we extend the ideas of CP-ELM and DP-ELM to sparsify RELM with two schemes, yielding CP-RELM and DP-RELM. To confirm the effectiveness and feasibility of the proposed algorithms, eleven benchmark data sets divided into two groups are utilized to do experiments in Section 5. Finally, conclusions follow.