Robust extreme learning machine
Introduction
Extreme learning machine (ELM) is an interesting learning algorithm proposed by Huang et al. [1]. It works on a simple structure named single-hidden layer feedforward neural networks (SLFNs). It randomly applies computational hidden nodes. This mechanism is different from the conventional learning of SLFNs. It provides a good generalization and a highly accurate learning solution for both classification and regression problems [2], [3]. Huang et al. mentioned that ELM yielded better performance than other conventional learning algorithms in application with higher noise [2]. ELM also has an extremely fast learning speed compared to traditional gradient-based algorithms. Furthermore, ELM technique successfully overcame the difficulty of the curse of dimensionality [4]. For regression problems, ELM has been widely applied in many real-world regression problems such as terrain models [5], prediction of melting points of organic compounds [6], sales forecasting of fashion retailing [7], modeling permeability prediction [8], etc. However, the computational robustness and the outlier robustness problems of ELM are not widely mentioned. The computational robustness is a computing ability of ELM to compute output weights from a formula like even if the hidden layer output matrix H is not full column rank or ill-conditioned. The ordinary ELM solved this problem by using the Singular Value Decomposition (SVD) approach [9]. However, the shortcoming of SVD is that it is very slow whenever the computed data set is large. The outlier robustness problem may occur whenever the training data is contaminated with outliers, which may result in ELM producing a poor and unreliable solution. An outlier is a sample with an error or noise due to human or device error that is outstanding and far away from other regular samples [10].
Wang et al. [11] proposed the effective extreme learning machine (EELM) to improve the computational robustness of ELM. However, EELM is limited with an activation function with one peak such as Gaussian radial basis function. Yuan et al. [12] proposed an optimization approximation solution algorithm for ELM. The is reformed to to obtain a solution based on the rank of H. Its advantage is that it can be used with H that is either full rank or not full rank. Huynh et al. [13] proposed ELM based on the weighted least squares (WLSs) scheme for reducing the effects of outliers in regression problems by using the penalty weights based on the norm of residuals. Their result showed a promising improvement in the outlier robustness of ELM. Deng et al. [14] proposed a regularized ELM to increase the generalization of ELM. They showed that ELM was seriously affected by outliers. However, the study into the effects of outliers in ELM is only in its infancy.
Thus, reducing the effects of not full column rank or ill-condition of the hidden layer output matrix H and the influence of outliers are extremely important and necessary. Therefore, this paper will address them simultaneously. The computational robustness problem is solved by replacing SVD with the Extended Complete Orthogonal Decomposition (ECOD). Three new iterative algorithms for ELM are proposed to improve the outlier robustness of ELM. They require robust inverse matrix computing. Accordingly, ECOD via ECODLS is also used for overcoming the not full column rank problem and the ill-conditioned matrix problem as well as the speeding up the training time of the common sub-problem, , in the three proposed algorithms.
The remainder of this paper is organized into five sections. The second section describes the background knowledge of ELM, the iteratively reweighted least squares (IRWLSs), the multivariate least-trimmed squares (MLTSs), and the one-step reweighted MLTS (RMLTS). The third section discusses the new proposed algorithms. The fourth section covers the experiments and results, and the fifth section is the conclusion and discussion. Finally, the Appendix describes ECOD and the implementation of ECODLS.
Section snippets
ELM and existing algorithms
This section will briefly describe outlier definition, ELM, IRWLS, MLTS, and RMLTS. IRWLS, MLTS, and RMLTS are the three algorithms that will be improved and used as the basis for the proposed algorithms to improve the robust property of ELM.
Robust extreme learning machine
The proposed algorithms to improve the outlier robustness characteristic of ELM are described in this section. These are the iteratively reweighted least squares extreme learning machine (IRWLS-ELM), the multivariate least-trimmed squares extreme learning machine (MLTS-ELM) and the reweighted multivariate least-trimmed squares extreme learning machine (RMLTS-ELM). However, there are several steps in the algorithms that , where , , , must be solved. The ECODLS will be
Experimental results
The purpose of the experimental design is to compare (a) the computational robustness of ECOD and SVD in ELM and (b) the outlier robustness of the proposed algorithms, IRWLS-ELM, MLTS-ELM, and RMLTS-ELM to the ordinary ELM (OLS-ELM) and the minimax probability machine regression (MPMR). The function pinv of MATLAB contributed to the output weights computing of OLS-ELM. MPMR has been used to successfully detect outliers [30]. It can predict whether the output of a regression problem according to
Conclusion
This paper solves the two robustness issues of ELM. The first issue is solved by improving the computational robustness of common sub-problems like using ECOD via ECODLS. The second is by proposing outlier robustness enhancement using three new algorithms: IRWLS-ELM, MLTS-ELM, and RMLTS-ELM. Our experimental results can be summarized as follows:
- (a)
ECOD can increase the computational robustness of ELM and produces more reliable and faster computing times, in case the large data set, than SVD.
- (b)
Acknowledgments
This work was partially supported by the Higher Education Research Promotion and National Research University Project of Thailand, Office of the Higher Education Commission, through the Cluster of Research to Enhance the Quality of Basic Education and Department of Computer Science, Faculty of Science, Khon Kaen University. This research is a part of the Computational Science Research Group (COSRG), Faculty of Science, Khon Kaen University (COSRG-SCKKU). We would also thank Mr. James McCloskey
Punyaphol Horata received the B.Sc. degree in Mathematics from Khon Kaen University, and M.S. degree in Computer Science from Chulalongkorn University, Thailand. He is currently a Ph.D. student at Khon Kaen University, Thailand. His research interests include machine learning, soft computing, and pattern recognition.
References (37)
- et al.
Extreme learning machine: theory and applications
Neurocomputing
(2006) - et al.
Upper integral network with extreme learning mechanism
Neurocomputing
(2011) - et al.
Sales forecasting using extreme learning machine with applications in fashion retailing
Decision Support Syst.
(2008) - et al.
A study on effectiveness of extreme learning machine
Neurocomputing
(2011) - et al.
Optimization approximation solution for regression problem based on extreme learning machine
Neurocomputing
(2011) - et al.
Enhanced random search based incremental extreme learning machine
Neurocomputing
(2008) - et al.
Partial Lanczos extreme learning machine for single-output regression problems
Neurocomputing
(2009) - et al.
Nox emission modelling using the iteratively reweighted least-square procedures
Int. J. Electr. Power Energy Syst.
(1995) - et al.
The multivariate least-trimmed squares estimator
J. Multivariate Anal.
(2008) - et al.
Meta-metric evaluation of e-commerce-related metrics
Electron. Notes Theor. Comput. Sci.
(2009)
Extreme learning machines: a survey
Int. J. Mach. Learn. Cybern.
Extreme learning machine for regression and multiclass classification
IEEE Trans. Syst. Man Cybern.: Part B
A new machine learning paradigm for terrain reconstruction
IEEE Geosci. Remote Sensing Lett.
Prediction of melting points of organic compounds using extreme learning machines
Ind. Eng. Chem. Res.
Matrix Computations
Robust Statistics Theory and Methods
Weighted least squares scheme for reducing effects of outliers in regression based on extreme learning machine
Int. J. Digital Content Technol. Appl. (JDCTA)
Cited by (156)
Elastic-net based robust extreme learning machine for one-class classification
2023, Signal ProcessingDynamic modeling of Boiler drum using nonlinear system identification approach
2023, Measurement: SensorsAmbient intelligence-based multimodal human action recognition for autonomous systems
2023, ISA TransactionsBig multi-step ship motion forecasting using a novel hybrid model based on real-time decomposition, boosting algorithm and error correction framework
2022, Ocean EngineeringCitation Excerpt :Zhang et al. (2020) also make progress in ship motion prediction by using the ELM model. Unfortunately, it is mentioned in the literature (Horata et al., 2013) that ELM models are vulnerable to outliers in the training data. The outlier robust extreme learning machine (ORELM) model shows remarkable performance in dealing with outliers while maintaining the above advantages (Zhang and Luo, 2015).
Machine learning techniques and data for stock market forecasting: A literature review
2022, Expert Systems with ApplicationsSmart grid stability prediction using genetic algorithm-based extreme learning machine
2022, Electric Power Systems Resiliency: Modelling, Opportunity and Challenges
Punyaphol Horata received the B.Sc. degree in Mathematics from Khon Kaen University, and M.S. degree in Computer Science from Chulalongkorn University, Thailand. He is currently a Ph.D. student at Khon Kaen University, Thailand. His research interests include machine learning, soft computing, and pattern recognition.
Sirapat Chiewchanwattana graduated in Statistics from Khon Kaen University, Thailand. She received her M.Sc. in Computer Science from the National Institute of Development Administration, Thailand and Ph.D. in Computer Science from Chulalongkorn University in 2007. She works as an Assistant Professor in the department of Computer Science at Khon Kaen University, Thailand and joined in research group in Intelligence System and Machine Learning. Her research interests are in neural networks, soft computing, and pattern recognition.
Khamron Sunat graduated in chemical engineering in Chulalongkorn University, Thailand, in 1989. He received his M.Sc. in Computational Science in 1998 and Ph.D. in Computer Science from Chulalongkorn University in 2004. He now works as a lecturer in the Department of Computer Science at Khon Kaen University, Thailand and joined in research group in Intelligence System and Machine Learning. His research interests are in Neural Networks, Pattern Recognition, Computer Vision, Soft Computing, Fuzzy Systems, Evolutionary Computing and Learning, Optical Computing.