Elsevier

Neurocomputing

Volume 174, Part A, 22 January 2016, Pages 368-374
Neurocomputing

A-ELM: Adaptive Distributed Extreme Learning Machine with MapReduce

https://doi.org/10.1016/j.neucom.2015.01.094Get rights and content

Abstract

Due to the outstanding advantage, such as generalization performance and fast convergence, Extreme Learning Machine (ELM) and its variants have been widely used for many applications. The distributed ELM with MapReduce could handle large-scale training dataset efficiently, but how to cope with its updated hidden nodes number which aims to get the higher accuracy is still a challenging task. In this paper, we propose a novel Adaptive Distributed Extreme Learning Machine with MapReduce (A-ELM). It could overcome the weakness of ELM in learning massive training dataset for updating hidden nodes number. Firstly, we found that through partial adjustment of incremental hidden nodes and decremental hidden nodes, matrix multiplication (the most computation-expensive part in A-ELM) can be calculated. Next, A-ELM based on MapReduce framework is proposed. A-ELM first calculates the intermediate matrix multiplications of the updated hidden nodes subset, and then update the matrix multiplications by modifying the old matrix multiplications with the intermediate ones. Then, based on the updated matrix multiplications, there could obtain the corresponding new output weight vector with centralized computing. Therefore, it is effective for learning large scale training dataset, in which the hidden nodes update rapidly. Finally, we verify the effectiveness and efficiency of our proposed A-ELM, using synthetic data with extensive experiments, in learning updated hidden nodes.

Introduction

With the rapid development of information technology, data has attracted more and more attentions. In recent years, with the development of the Internet, Cloud Computing, Mobile Internet and Internet of Things, the amount of information has the growth trend exponentially. Thus, big data [1] becomes a hot technology in many machine learning applications. Generally, the larger the training sample, the better the classifier. However, due to the characteristics of the ‘Value’ of big data [2], high commercial value and low density value, the training data will be constantly updated. With the change of date, the existed machine learning model will not be so accurate for the updated training dataset, thereby reducing the classification accuracy. Thus, it is necessary to change the classification model to achieve the classification accuracy needed, according to the updated training dataset.

Due to the characteristics of excellent generalization performance, rapid training speed and less human intervene, Extreme Learning Machine (ELM) [3], [4], [5], [6], [7], [8] has recently attracted increasing attention from researchers [9]. Because of its advantages, ELM can be applied in many fields and display a significant result [10], [11], [12], [13], [14], [15], [16], [17], [18]. With the improvement of ELM, distributed ELM (ELM [19]) based on MapReduce [20], [21], [22] can handle a great volume of training dataset. However, it is quite common in big data classifications that the training dataset needs to be updated. A simple and direct way is to retraining the ELM using the whole training dataset. Obviously, this kind of method is time-consuming. Therefore, an Elastic ELM (E2LM) [23] is proposed to improve ELM with the abilities of incremental learning, decremental learning, and correctional learning. But there is another problem that the classification model may not be so accurate when the dateset has changed without changing the number of hidden nodes. Thus, the model could be adjusted to adapt updated dataset so as to achieve higher classification accuracy.

A simple and direct way is to re-calculate by ELM [19] with updated numbers of hidden nodes using the whole training dataset. Obviously, it is time-consuming with this kind of method, because the training dataset is too large. So it is necessary to improve the ELM [19] algorithm to adjust the classification model to support the functionalities such as incremental hidden nodes and decremental hidden nodes. There is a great overlap between the old hidden nodes and the new ones, if the overlapped information can be used, the cost of training time with new classification model will reduce greatly. Therefore, in this paper, an Adaptive Distributed Extreme Learning Machine is proposed to improve ELM [19] with the abilities of incremental hidden nodes and decremental hidden nodes. The contributions of this paper can be summarized as follows:

  • 1.

    We prove theoretically that the most expensive computation part in ELM can be incrementally and decrementally calculated, which indicate that ELM can be extended to support incremental hidden nodes and decremental hidden nodes.

  • 2.

    A novel Adaptive Distributed Extreme Learning Machine based on MapReduce framework is proposed, which can enhance the training performance of ELM for handling the updated hidden nodes.

  • 3.

    Last but not least, our extensive experimental studies using synthetic data show that our proposed A-ELM can learn updated hidden nodes with dataset effectively, which can fulfill the requirements of classification accuracy in many big data classification applications.

The remainder of the paper is organized as follows. There is brief introduction of traditional ELM and distributed ELM in Section 2. Section 3 presents the theoretical foundation and the computational details of the proposed A-ELM approach. The experimental results are reported in Section 4 to show the effectiveness and efficiency of our A-ELM approach. Finally, we conclude this paper in Section 5.

Section snippets

Background

In this section, we describe the background for our work, which includes a brief overview of traditional ELM and distributed ELM (ELM).

Adaptive distributed ELM

In this section, we first explain the derivation of Adaptive Distributed ELM (A-ELM) in detail. And then, we present the computational details of our proposed A-ELM.

Performance evaluation

In this section, the performance of our Adaptive Distributed ELM is evaluated in detail with various experimental settings. We first describe the platform used in our experiments in Section 4.1. Then we present and discuss the experimental results of the evaluation in Section 4.2.

Conclusions

In order to cover the shortage of ELM whose learning ability is weak to the updated hidden nodes with large-scale training dataset, we propose a novel Adaptive Distributed Extreme Learning Machine based on MapReduce framework. A-ELM can support incremental and decremental hidden nodes learning of massive training dataset. Specifically, after analyzing the property of ELM adequately, we found that the most computation-expensive part can be incrementally and decrementally calculated. Then, the

Acknowledgments

This research was partially supported by the National Natural Science Foundation of China under Grant nos. 61402089 and 61472069; the National High Technology Research and Development Plan (863 Plan) under Grant no. 2012AA02A606; and the Fundamental Research Funds for the Central Universities under Grant no. N130404014.

Junchang Xin received B.Sc., M.Sc and Ph.D. in Computer Science and Technology from the Northeastern University, P.R. China, in July 2002, March 2005, and July 2008, respectively. He visited National University of Singapore (NUS) as post-doctoral visitor (April 2010 – April 2011). He is currently an associate professor in the Department of Computer Science, Northeastern University, P.R. China. His research interests include big data, cloud computing, data management over wireless sensor network

References (23)

  • D. Garlasu, A big data implementation based on grid computing, In: Proceedings of the Roedunet International Conference...
  • Cited by (19)

    • Collaborative extreme learning machine with a confidence interval for P2P learning in healthcare

      2019, Computer Networks
      Citation Excerpt :

      Even though [33] explored incremental ELM to address the above limitations in OS-ELM and fully supports OS-learning, instead of considering P2P or distributed learning which is significant to improve diagnostic accuracy by sharing knowledge, the above-mentioned techniques only enhance the local models and focus on multiclass classification. In terms of distributed learning, [14] and [15] employ MapReduce (a common framework for distributed computing) to speed up the ELM training process with a large number of samples. Their systems contain a master server and several slave servers.

    • Fast and peer-to-peer vital signal learning system for cloud-based healthcare

      2018, Future Generation Computer Systems
      Citation Excerpt :

      For Cloud computing, a centralized algorithm such as MapReduce is typically used in varieties of studies. Among them, distributed training methods with ELM [39,40] and HMM [7] using MapReduce have been proposed to accelerate the training process with the computational power of Cloud. In addition, some ensemble based models [41–43] based on MapReduce have been also adopted to avoid the over-fitting problem and to improve the accuracy of the classifiers.

    • HB-File: An efficient and effective high-dimensional big data storage structure based on US-ELM

      2017, Neurocomputing
      Citation Excerpt :

      Finally, we conclude this paper in Section 6. Nowadays, Extreme Learning Machine (ELM) [4] and its variants [5–12] have the characteristics of excellent generalization performance, rapid training speed and little manual intervention, which have attracted increasing attention from more and more researchers. ELM is originally designed for single hidden-layer feedforward neural networks (SLFNs [13]) and is then extended to the “generalized” SLFNs.

    • Distributed extreme learning machine with alternating direction method of multiplier

      2017, Neurocomputing
      Citation Excerpt :

      Based on the analysis above, how to implement extreme learning machine with respect to the data set which located in different machines becomes a key problems. In previous work, some parallel or distributed extreme learning machine have been implemented to meet the challenge of large-scale data set [11,12]. For example, Q.

    • Distributed learning for feedforward neural networks with random weights using an event-triggered communication scheme

      2017, Neurocomputing
      Citation Excerpt :

      Such large-scale datasets are being collected and accumulated at a dramatic pace, resulting in the emergence of new research in the field of large-scale machine learning. Most of existing work in this area has focused on addressing how to seek fast and efficient solutions in a parallel processing manner, e.g., MapReduce [8], where huge datasets are allocated to different agents (or network nodes) for processing and the local results are sent back to the fusion center for a final decision [9–12]. However, parallel techniques are confronted with significant challenges in some occasions: First, the datasets may be present in physically, geographically or logically a distributed manner where there is no any centralized authority.

    View all citing articles on Scopus

    Junchang Xin received B.Sc., M.Sc and Ph.D. in Computer Science and Technology from the Northeastern University, P.R. China, in July 2002, March 2005, and July 2008, respectively. He visited National University of Singapore (NUS) as post-doctoral visitor (April 2010 – April 2011). He is currently an associate professor in the Department of Computer Science, Northeastern University, P.R. China. His research interests include big data, cloud computing, data management over wireless sensor network and uncertain data management. He has published more than 40 research papers.

    Zhiqiong Wang received her M.Sc and Ph.D. in Computer Science and Technology from the Northeastern University, P.R. China, in 2008 and 2014, respectively. She visited the National University of Singapore (NUS) in 2010 and the Chinese University of Hong Kong (CUHK) in 2013 as the academic visitor. Currently, she is an associate professor of the Sino-Dutch Biomedical and Information Engineering School of Northeastern University. Her main research interests are biomedical, biological data processing, cloud computing, and machine learning. She has published more than 30 papers in journals and conferences.

    Luxuan Qu received her Bachelor׳s degree in automatic from Northeast Dianli University in 2010. Currently, she is a master student of the Sino-Dutch Biomedical and Information Engineering School of Northeastern University. Her main research interests are biological data processing, machine learning, and cloud computing.

    Ge Yu was born in Dalian in 1962. He received his B.E. degree and M.E degree in Computer Science from Northeastern University of China in 1982 and 1986, respectively, Ph.D. degree in Computer Science from Kyushu University of Japan in 1996. He has been a professor at Northeastern University of China since 1996. He is a member of IEEE, ACM, and a senior member of CCF. His research interests include database theory and technology, distributed and parallel systems, cyber-physical systems, real time complex event processing, embedded systems, and Web information security.

    Yan Kang received his B.E. of Information and Control Engineering and M.E. of Biomedical Engineering from Xi’an Jiao tong University in 1987 and 1990, respectively, and Ph.D. degree from University of Erlangen-Nürnberg in Germany in 2002. He worked as a postdoctoral fellow in Image Guidance Lab, Medical Center of Stanford University. Currently, he is a professor and th dean of Sino-Dutch Biomedical and Information Engineering School, Northeastern University of China. His research interests include medical imaging, intelligent auxiliary medical, computer-aided diagnosis and treatment, and computer simulations. He developed 6 computer Intelligent Assistant software products, 11 patents, 1 monograph, 1 translation works and 1 item of appraisal results, and published more than 70 papers.

    View full text