Partial Lanczos extreme learning machine for single-output regression problems

Abstract

There are two problems preventing the further development of extreme learning machine (ELM). First, the ill-conditioning of hidden layer output matrix reduces the stability of ELM. Second, the complexity of singular value decomposition (SVD) for computing Moore–Penrose generalized inverse limits the learning speed of ELM. For these two problems, this paper proposes the partial Lanczos ELM (PL-ELM) which employs the hybrid of partial Lanczos bidiagonalization and SVD to compute output weights. Experimental results indicate that, compared with ELM, PL-ELM not only effectively improves the stability and generalization performance but also raises the learning speed.

Introduction

Recently, Huang et al. [1], [2] have proposed a novel learning algorithm for single hidden layer feedforward networks (SLFNs) named extreme learning machine (ELM). In ELM, input weights and hidden layer bias are randomly chosen, and output weights are analytically determined based on the Moore–Penrose generalized inverse of hidden layer output matrix. ELM provides better generalization performance with extremely fast learning speed than gradient-based learning algorithms. In addition, ELM avoids many difficulties faced by gradient-based learning methods such as stopping criteria, learning rate, learning epochs and local minima [1], [3]. Depending upon these advantages, ELM has been successfully applied in many areas, such as classification [4], function approximation [5], [6], nontechnical loss analysis [7], terrain reconstruction [8] and protein structure prediction [9].

In order to enhance the performance of ELM, many improved models of ELM have been proposed. We roughly summarize these improved models of ELM into four types: incremental type, optimization type, replacement type and ensemble type. The incremental type of ELM creates new hidden layer neurons one by one according to certain criteria, which is very suitable for stream data [10], [11], [12]. The optimization type of ELM employs some techniques, such as evolutionary algorithm and linear programming, to tune input weights and hidden layer bias and optimize the network structure; it achieves good generalization performance and much more compact network structure [13], [14], [15]. The replacement type of ELM replaces the activation functions of ELM (sigmoid and RBF) with sine and cosine functions (or other functions), which is helpful to improve the accuracy and the convergence rate for the problem of function approximation [6]. In the ensemble type of ELM, different ELMs are trained by disjoint subsets of data, but they can share the same hidden layer neurons [5], [16].

Although these models improve the performance of ELM to a certain degree, there still exists great development space for the stability and the learning speed of ELM. We firstly analyze the critical factor influencing the stability of ELM.

As rigorously proven by Huang et al. [1], input weights and hidden layer bias of SLFNs can be randomly chosen without tuning, and then the hidden layer output matrix H of SLFNs can remain unchanged in the whole learning process. Thus, training a SLFN is equivalent to finding the minimal Euclidean norm solution β of the linear system $\mathrm{H}\beta =\mathrm{T}$ [1]:

$$\beta ={\mathrm{H}}^{†}\mathrm{T}\mathrm{such}\mathrm{that}\beta =\underset{\beta }{\arg \min }\parallel \mathrm{H}\beta -\mathrm{T}{\parallel }_2$$

where β denotes the output weight matrix of the trained ELM, ${\mathrm{H}}^{†}$ denotes the Moore–Penrose generalized inverse of the hidden layer output matrix H, T denotes the target vector. The detailed specification of these variables is given in [1].

ELM computes the Moore–Penrose generalized inverse ${\mathrm{H}}^{†}$ in Eq. (1) based on the singular value decomposition (SVD) of H [17]. The SVD of $\mathrm{H}\in {\mathbb{R}}^{N\times n},N⩾n$, is given by

$$\mathrm{H}=U\Sigma V^T=\sum _{i=1}^n{u}_i{\sigma }_i{v}_i^T$$

where $U=(u_1,\dots ,u_n)$, $V=(v_1,\dots ,v_n)$, $\Sigma =\mathrm{diag}({\sigma }_1,\dots ,{\sigma }_n)$, ${\sigma }_i\in \Sigma ,i=1,\dots ,n$ are singular values of H with the order ${\sigma }_1⩾\cdots ⩾{\sigma }_n⩾0$, and the rank r of H equals the number of strictly positive singular values, i.e. ${\sigma }_r>{\sigma }_{r+1}=\cdots ={\sigma }_n=0$.

Based on Eq. (2), the Moore–Penrose generalized inverse ${\mathrm{H}}^{†}$ in Eq. (1) is computed by

$${\mathrm{H}}^{†}=V{\Sigma }^{†}{U}^T=\sum _{i=1}^r\frac{v_i{u}_i^T}{{\sigma }_i}$$

and the output weight matrix of ELM is computed by

$$\beta ={\mathrm{H}}^{†}\mathrm{T}=\sum _{i=1}^r\frac{v_i{u}_i^T}{{\sigma }_i}\mathrm{T}$$

In practical applications, the target vector T usually contains a certain degree of perturbations. Let e denote the perturbation in T and $\tilde{\mathrm{T}}=\mathrm{T}+e$ denote the perturbed target vector. The perturbed output weight matrix $\tilde{\beta }$ of ELM can be computed by

$$\tilde{\beta }={\mathrm{H}}^{†}\tilde{\mathrm{T}}={\mathrm{H}}^{†}\mathrm{T}+{\mathrm{H}}^{†}e=\sum _{i=1}^r\frac{v_i{u}_i^T}{{\sigma }_i}\mathrm{T}+\sum _{i=1}^r\frac{v_i{u}_i^T}{{\sigma }_i}e$$

When the hidden layer output matrix H has a very large condition number (i.e. H tending to be ill-conditioned), there always exist some very small (positive and approximate to zero) singular values in H. From the two summation items in the right hand of Eq. (5) we can find that, when divided by small singular values, the output weights become very large and tend to be greatly influenced by the perturbation e. The large output weights also weaken the generalization capability of ELM, because the trained network will behave very different if test data change but slightly away from the training data. The ill-conditioning of hidden layer output matrix is the critical factor influencing the stability of ELM.

We then analyze the critical factor limiting the learning speed of ELM as follows. The learning time of ELM is mainly consumed by computing the Moore–Penrose generalized inverse ${\mathrm{H}}^{†}$ [1]. As mentioned above (see Eqs. (2), (3), (4)), computing ${\mathrm{H}}^{†}$ requires the SVD of H. For matrix $\mathrm{H}\in {\mathbb{R}}^{N\times n}$, the computational complexity of SVD is $O(4N{n}^2+8n^3)$ [18]. When the number of hidden layer neurons (i.e. n) becomes large, the computational complexity of SVD will significantly rise. Although some methods are proposed to compact the network structure of ELM [13], [14], they make ELM lose its property of determining network structure completely independent from training data. The large computational complexity of SVD is the critical factor limiting the learning speed of ELM.

To improve the stability of ELM, it is necessary to incorporate some regularization methods for reducing the influences of ill-conditioning and perturbations. Truncated SVD (TSVD) and Tikhonov regularization [19], [20] are two kinds of widely used regularization methods. As proven by [19], [21], TSVD can provide similar results with Tikhonov regularization. TSVD only focuses on the contributions associated with the k largest singular values and effectively avoids the adverse effects caused by the smallest singular values. According to TSVD, the Moore–Penrose generalized inverse of H can be computed by

$${\mathrm{H}}^{†}=\sum _{i=1}^{\kappa }\frac{v_i{u}_i^T}{{\sigma }_i},\kappa ⩽r=\mathrm{rank}(\mathrm{H})⩽n$$

where κ is the truncation number of TSVD for H, usually $\kappa ⪡n$.

To improve the learning speed of ELM, it is necessary to reduce the computational complexity of SVD. Although TSVD provides more stable solutions than SVD, it still depends on the results of SVD and has the similar computational complexity with SVD [22]. Lanczos bidiagonalization (LBD) is an efficient iterative method for computing the SVD of large and ill-conditioned matrices [23], [24], [25]. Given the appropriate number of iterations, we can conduct partial LBD for a matrix. Partial LBD not only makes the computation of SVD fairly inexpensive but also provides good approximation to the singular triplets associated with the largest singular values of the matrix [26]. Partial LBD has been applied to the computation of many regularization methods, such as Tikhonov regularization and TSVD [27], [28], [29].

In this paper, we propose an enhanced ELM, called partial Lanczos ELM (PL-ELM), which computes the output weights based on the hybrid of partial LBD and SVD. PL-ELM first implements partial LBD to the hidden layer output matrix H so that the linear system $\mathrm{H}\beta =\mathrm{T}$ can be projected onto a small-size Krylov subspace, then PL-ELM determines the approximate solution of output weights from the Krylov subspace. Since the dimension of the Krylov subspace is usually much smaller than that of hidden layer output matrix, PL-ELM can significantly reduce the computational complexity compared with ELM. On the other hand, the results of PL-ELM are very similar to those achieved by directly applying TSVD to the linear system $\mathrm{H}\beta =\mathrm{T}$. Currently, PL-ELM mainly deals with the single-output regression problem. We validate the performance of PL-ELM using several benchmark regression data sets.

This paper is organized as follows. Section 2 briefly reviews the partial LBD algorithm; Section 3 proposes the PL-ELM algorithm and then analyzes the computational complexity, the relative perturbation bound and the parameter choice of the proposed PL-ELM algorithm; Section 4 shows experimental results and discussions; Section 5 contains the conclusion and the consideration of future research.