Elsevier

Neurocomputing

Volume 102, 15 February 2013, Pages 31-44
Neurocomputing

Robust extreme learning machine

https://doi.org/10.1016/j.neucom.2011.12.045Get rights and content

Abstract

The output weights computing of extreme learning machine (ELM) encounters two problems, the computational and outlier robustness problems. The computational problem occurs when the hidden layer output matrix is a not full column rank matrix or an ill-conditioned matrix because of randomly generated input weights and biases. An existing solution to this problem is Singular Value Decomposition (SVD) method. However, the training speed is still affected by the large complexity of SVD when computing the Moore–Penrose (MP) pseudo inverse. The outlier robustness problem may occur when the training data set contaminated with outliers then the accuracy rate of ELM is extremely affected. This paper proposes the Extended Complete Orthogonal Decomposition (ECOD) method to solve the computational problem in ELM weights computing via ECODLS algorithm. And the paper also proposes the other three algorithms, i.e. the iteratively reweighted least squares (IRWLS-ELM), ELM based on the multivariate least-trimmed squares (MLTS-ELM), and ELM based on the one-step reweighted MLTS (RMLTS-ELM) to solve the outlier robustness problem. However, they also encounter the computational problem. Therefore, the ECOD via ECODLS algorithm is also used successfully in the three proposed algorithms. The experiments of regression problems were conducted on both toy and real-world data sets. The outlier types are one-sided and two-sided outliers. Each experiment was randomly contaminated with outliers, of one type only, with 10%, 20%, 30%, 40%, and 50% of the total training data size. Meta-metrics evaluation was used to measure the outlier robustness of the proposed algorithms compared to the existing algorithms, i.e. the minimax probability machine regression (MPMR) and the ordinary ELM. The experimental results showed that ECOD can effectively replace SVD. The ECOD is robust to the not full column rank or the ill-conditional problem. The speed of the ELM training using ECOD is also faster than the ordinary training algorithm. Moreover, the meta-metrics measure showed that the proposed algorithms are less affected by the increasing number of outliers than the existing algorithms.

Introduction

Extreme learning machine (ELM) is an interesting learning algorithm proposed by Huang et al. [1]. It works on a simple structure named single-hidden layer feedforward neural networks (SLFNs). It randomly applies computational hidden nodes. This mechanism is different from the conventional learning of SLFNs. It provides a good generalization and a highly accurate learning solution for both classification and regression problems [2], [3]. Huang et al. mentioned that ELM yielded better performance than other conventional learning algorithms in application with higher noise [2]. ELM also has an extremely fast learning speed compared to traditional gradient-based algorithms. Furthermore, ELM technique successfully overcame the difficulty of the curse of dimensionality [4]. For regression problems, ELM has been widely applied in many real-world regression problems such as terrain models [5], prediction of melting points of organic compounds [6], sales forecasting of fashion retailing [7], modeling permeability prediction [8], etc. However, the computational robustness and the outlier robustness problems of ELM are not widely mentioned. The computational robustness is a computing ability of ELM to compute output weights from a formula like Hβ=T even if the hidden layer output matrix H is not full column rank or ill-conditioned. The ordinary ELM solved this problem by using the Singular Value Decomposition (SVD) approach [9]. However, the shortcoming of SVD is that it is very slow whenever the computed data set is large. The outlier robustness problem may occur whenever the training data is contaminated with outliers, which may result in ELM producing a poor and unreliable solution. An outlier is a sample with an error or noise due to human or device error that is outstanding and far away from other regular samples [10].

Wang et al. [11] proposed the effective extreme learning machine (EELM) to improve the computational robustness of ELM. However, EELM is limited with an activation function with one peak such as Gaussian radial basis function. Yuan et al. [12] proposed an optimization approximation solution algorithm for ELM. The Hβ=T is reformed to HTHβ=HTT to obtain a solution based on the rank of H. Its advantage is that it can be used with H that is either full rank or not full rank. Huynh et al. [13] proposed ELM based on the weighted least squares (WLSs) scheme for reducing the effects of outliers in regression problems by using the penalty weights based on the norm of residuals. Their result showed a promising improvement in the outlier robustness of ELM. Deng et al. [14] proposed a regularized ELM to increase the generalization of ELM. They showed that ELM was seriously affected by outliers. However, the study into the effects of outliers in ELM is only in its infancy.

Thus, reducing the effects of not full column rank or ill-condition of the hidden layer output matrix H and the influence of outliers are extremely important and necessary. Therefore, this paper will address them simultaneously. The computational robustness problem is solved by replacing SVD with the Extended Complete Orthogonal Decomposition (ECOD). Three new iterative algorithms for ELM are proposed to improve the outlier robustness of ELM. They require robust inverse matrix computing. Accordingly, ECOD via ECODLS is also used for overcoming the not full column rank problem and the ill-conditioned matrix problem as well as the speeding up the training time of the common sub-problem, Ax=b, in the three proposed algorithms.

The remainder of this paper is organized into five sections. The second section describes the background knowledge of ELM, the iteratively reweighted least squares (IRWLSs), the multivariate least-trimmed squares (MLTSs), and the one-step reweighted MLTS (RMLTS). The third section discusses the new proposed algorithms. The fourth section covers the experiments and results, and the fifth section is the conclusion and discussion. Finally, the Appendix describes ECOD and the implementation of ECODLS.

Section snippets

ELM and existing algorithms

This section will briefly describe outlier definition, ELM, IRWLS, MLTS, and RMLTS. IRWLS, MLTS, and RMLTS are the three algorithms that will be improved and used as the basis for the proposed algorithms to improve the robust property of ELM.

Robust extreme learning machine

The proposed algorithms to improve the outlier robustness characteristic of ELM are described in this section. These are the iteratively reweighted least squares extreme learning machine (IRWLS-ELM), the multivariate least-trimmed squares extreme learning machine (MLTS-ELM) and the reweighted multivariate least-trimmed squares extreme learning machine (RMLTS-ELM). However, there are several steps in the algorithms that Ax=b, where ARn×p, xRp×q, bRn×q,q1, must be solved. The ECODLS will be

Experimental results

The purpose of the experimental design is to compare (a) the computational robustness of ECOD and SVD in ELM and (b) the outlier robustness of the proposed algorithms, IRWLS-ELM, MLTS-ELM, and RMLTS-ELM to the ordinary ELM (OLS-ELM) and the minimax probability machine regression (MPMR). The function pinv of MATLAB contributed to the output weights computing of OLS-ELM. MPMR has been used to successfully detect outliers [30]. It can predict whether the output of a regression problem according to

Conclusion

This paper solves the two robustness issues of ELM. The first issue is solved by improving the computational robustness of common sub-problems like Ax=b using ECOD via ECODLS. The second is by proposing outlier robustness enhancement using three new algorithms: IRWLS-ELM, MLTS-ELM, and RMLTS-ELM. Our experimental results can be summarized as follows:

  • (a)

    ECOD can increase the computational robustness of ELM and produces more reliable and faster computing times, in case the large data set, than SVD.

  • (b)

Acknowledgments

This work was partially supported by the Higher Education Research Promotion and National Research University Project of Thailand, Office of the Higher Education Commission, through the Cluster of Research to Enhance the Quality of Basic Education and Department of Computer Science, Faculty of Science, Khon Kaen University. This research is a part of the Computational Science Research Group (COSRG), Faculty of Science, Khon Kaen University (COSRG-SCKKU). We would also thank Mr. James McCloskey

Punyaphol Horata received the B.Sc. degree in Mathematics from Khon Kaen University, and M.S. degree in Computer Science from Chulalongkorn University, Thailand. He is currently a Ph.D. student at Khon Kaen University, Thailand. His research interests include machine learning, soft computing, and pattern recognition.

References (37)

  • G.-B. Huang et al.

    Extreme learning machines: a survey

    Int. J. Mach. Learn. Cybern.

    (2011)
  • G.-B. Huang et al.

    Extreme learning machine for regression and multiclass classification

    IEEE Trans. Syst. Man Cybern.: Part B

    (2012)
  • C.-W.T. Yeu et al.

    A new machine learning paradigm for terrain reconstruction

    IEEE Geosci. Remote Sensing Lett.

    (2006)
  • A.U. Bhat et al.

    Prediction of melting points of organic compounds using extreme learning machines

    Ind. Eng. Chem. Res.

    (2008)
  • S.O. Olatunji, A. Selamat, A.A.A.R. Abdul, Modeling permeability prediction using extreme learning machines, in:...
  • G.H. Golub et al.

    Matrix Computations

    (1996)
  • R. Maronna et al.

    Robust Statistics Theory and Methods

    (2006)
  • H.T. Huynh et al.

    Weighted least squares scheme for reducing effects of outliers in regression based on extreme learning machine

    Int. J. Digital Content Technol. Appl. (JDCTA)

    (2008)
  • Cited by (156)

    • Big multi-step ship motion forecasting using a novel hybrid model based on real-time decomposition, boosting algorithm and error correction framework

      2022, Ocean Engineering
      Citation Excerpt :

      Zhang et al. (2020) also make progress in ship motion prediction by using the ELM model. Unfortunately, it is mentioned in the literature (Horata et al., 2013) that ELM models are vulnerable to outliers in the training data. The outlier robust extreme learning machine (ORELM) model shows remarkable performance in dealing with outliers while maintaining the above advantages (Zhang and Luo, 2015).

    • Smart grid stability prediction using genetic algorithm-based extreme learning machine

      2022, Electric Power Systems Resiliency: Modelling, Opportunity and Challenges
    View all citing articles on Scopus

    Punyaphol Horata received the B.Sc. degree in Mathematics from Khon Kaen University, and M.S. degree in Computer Science from Chulalongkorn University, Thailand. He is currently a Ph.D. student at Khon Kaen University, Thailand. His research interests include machine learning, soft computing, and pattern recognition.

    Sirapat Chiewchanwattana graduated in Statistics from Khon Kaen University, Thailand. She received her M.Sc. in Computer Science from the National Institute of Development Administration, Thailand and Ph.D. in Computer Science from Chulalongkorn University in 2007. She works as an Assistant Professor in the department of Computer Science at Khon Kaen University, Thailand and joined in research group in Intelligence System and Machine Learning. Her research interests are in neural networks, soft computing, and pattern recognition.

    Khamron Sunat graduated in chemical engineering in Chulalongkorn University, Thailand, in 1989. He received his M.Sc. in Computational Science in 1998 and Ph.D. in Computer Science from Chulalongkorn University in 2004. He now works as a lecturer in the Department of Computer Science at Khon Kaen University, Thailand and joined in research group in Intelligence System and Machine Learning. His research interests are in Neural Networks, Pattern Recognition, Computer Vision, Soft Computing, Fuzzy Systems, Evolutionary Computing and Learning, Optical Computing.

    View full text