Skip to main content
SpringerLink
Log in

Quick extreme learning machine for large-scale classification

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

The extreme learning machine (ELM) is a method to train single-layer feed-forward neural networks that became popular because it uses a fast closed-form expression for training that minimizes the training error with good generalization ability to new data. The ELM requires the tuning of the hidden layer size and the calculation of the pseudo-inverse of the hidden layer activation matrix for the whole training set. With large-scale classification problems, the computational overload caused by tuning becomes not affordable, and the activation matrix is extremely large, so the pseudo-inversion is very slow and eventually the matrix will not fit in memory. The quick extreme learning machine (QELM), proposed in the current paper, is able to manage large classification datasets because it: (1) avoids the tuning by using a bounded estimation of the hidden layer size from the data population; and (2) replaces the training patterns in the activation matrix by a reduced set of prototypes in order to avoid the storage and pseudo-inversion of large matrices. While ELM or even the linear SVM cannot be applied to large datasets, QELM can be executed on datasets up to 31 million data, 30,000 inputs and 131 classes, spending reasonable times (less than 1 h) in general purpose computers without special software nor hardware requirements and achieving performances similar to ELM.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. The notation \(\{ a_{hi} \}_{hi=1}^{H,I}\) means \(h=1,\ldots ,H\) and \(i=1,\ldots ,I\).

  2. An exponent \(\alpha \ne 1\) may be used in the exponential, and alternative activation functions may be also used.

  3. https://archive.ics.uci.edu (Visited May, 2021).

  4. https://www.kaggle.com (Visited May, 2021).

  5. https://www.csie.ntu.edu.tw/~cjlin/libsvm (Visited May, 2021).

  6. https://www.csie.ntu.edu.tw/~cjlin/liblinear (Visited May, 2021).

  7. http://www.octave.org (Visited May, 2021).

References

  1. Afzal A, Nair N, Asharaf S (2021) Deep kernel learning in extreme learning machines. Patt Anal Appl 24(2):11–19

    Google Scholar 

  2. Ahmad I, Basheri M, Iqbal M, Rahim A (2018) Performance comparison of support vector machine, random forest, and extreme learning machine for intrusion detection. IEEE Access 6:33789–33795

    Google Scholar 

  3. Akram-Ali-Hammouri Z, Fernández-Delgado M, Cernadas E, Barro S (2021) Fast support vector classification for large-scale problems. IEEE T Pat Anal. https://doi.org/10.1109/TPAMI.2021.3085969

    Article  Google Scholar 

  4. Akusok A, Björk K, Miche Y, Lendasse A (2015) High-performance extreme learning machines: a complete toolbox for big data applications. IEEE Access 3:1011–1025

    Google Scholar 

  5. Cao F, Liu B, Park D (2013) Image classification based on effective extreme learning machine. Neurocomputing 102:90–97

    Google Scholar 

  6. Cao J, Lin Z (2013) Extreme learning machines on high dimensional and large data applications: a survey. Math Probl Eng 2015:1–13

    Google Scholar 

  7. Carletta J (1996) Assessing agreement on classification tasks: the kappa statistic. Comput Linguist 22(2):249–254

    Google Scholar 

  8. Chang C, Lin C (2011) LIBSVM: a library for support vector machines. ACM T Intel Syst Tech 2:1–27

    Google Scholar 

  9. Colton T (1974) Statistics in medicine. Little Brown and Co

    Google Scholar 

  10. Coppersmith D, Winograd S (1987) Matrix multiplication via arithmetic progressions. In: Proc ACM Sympos Theory Comput. pp. 1–6

  11. Courrieu P (2005) Fast computation of Moore-Penrose inverse matrices. Neu Inf Pro Lett Rev 8:25–29

    Google Scholar 

  12. Deng W, Bai Z, Huang G, Zheng Q (2016) A fast SVD-hidden-nodes based extreme learning machine for large-scale data analytics. Neural Netw 77:14–28

    MATH  Google Scholar 

  13. Dias M, de Sousa L, Neto A, Freire A (2018) Fixed-size extreme learning machines through simulated annealing. Neural Process Lett 48(1):135–151

    Google Scholar 

  14. Duan M, Li K, Liao X, Li K (2018) A parallel multiclassification algorithm for big data using an extreme learning machine. IEEE T Neur Net Lear 29(6):2337–2351

    MathSciNet  Google Scholar 

  15. Ertuğrul ÖF, Tağluk ME (2017) A fast feature selection approach based on extreme learning machine and coefficient of variation. Turk J Elec Eng Comput Sci 25(4):3409–3420

    Google Scholar 

  16. Fan R, Chang K, Hsieh C, Wang X, Lin C (2008) LIBLINEAR: a library for large linear classification. J Mach Learn Res 9:1871–1874

    MATH  Google Scholar 

  17. Hassan S, Khanesar M, Jaafar J, Khosravi A (2018) Optimal parameters of an ELM-based interval type 2 fuzzy logic system: a hybrid learning algorithm. Neural Comput Appl 29(4):1001–1014

    Google Scholar 

  18. He Q, Shang T, Zhuang F, Shi Z (2013) Parallel extreme learning machine for regression based on MapReduce. Neurocomputing 102:52–58

    Google Scholar 

  19. Hsu H, Chang C, Hsu C (2017) Chapter 6: extreme learning machine and its applications in big data processing. In: Big data analytics for sensor-network collected intelligence, Intelligent Data-Centric Systems, pp. 117–150. Academic Press

  20. Huang G, Liu T, Yang Y, Lin Z, Song S, Wu C (2015) Discriminative clustering via extreme learning machine. Neural Netw 70:1–8

    MATH  Google Scholar 

  21. Huang GB, Zhou H, Ding X, Zhang R (2011) Extreme learning machine for regression and multiclass classification. IEEE T Syst Man Cy B 42(2):513–529

    Google Scholar 

  22. Iosifidis A, Tefas A, Pitas I (2015) Regularized extreme learning machine for large-scale media content analysis. Procedia Comput Sci 53:420–427

    Google Scholar 

  23. Iosifidis A, Tefas A, Pitas I (2017) Approximate kernel extreme learning machine for large scale data classification. Neurocomputing 219:210–220

    Google Scholar 

  24. Khan K, Ratner E, Ludwig R, Lendasse A (2020) Feature bagging and extreme learning machines: machine learning with severe memory constraints. In: Proc Intl J Conf Neur Netw, pp. 1–7

  25. Köktürk-Güzel B, Beyhan S (2021) Symbolic regression based extreme learning machine models for system identification. Neural Proc Lett 53:1565–1578

    Google Scholar 

  26. Kongsorot Y, Horata P, Musikawan P (2020) An incremental kernel extreme learning machine for multi-label learning with emerging new labels. IEEE Access 8:46055–46070

    Google Scholar 

  27. Lan Y, Soh Y, Huang G (2009) Ensemble of online sequential extreme learning machine. Neurocomputing 72(13–15):3391–3395

    Google Scholar 

  28. Larrea M, Porto A, Irigoyen E, Barragán A, Andújar J (2009) Extreme learning machine ensemble model for time series forecasting boosted by PSO: application to an electric consumption problem. Neurocomputing 72(13–15):3391–3395

    Google Scholar 

  29. Li L, Wang G, Wu G, Zhang Q (2020) An experimental evaluation of extreme learning machines on several hardware devices. Neural Comput Appl 32:14385–14397

    Google Scholar 

  30. Liang N, Huang G, Saratchandran P, Sundararajan N (2006) A fast and accurate online sequential learning algorithm for feedforward networks. IEEE T Neural Networ 17(6):1411–1423

    Google Scholar 

  31. Liu X, Gao C, Li P (2012) A comparative analysis of support vector machines and extreme learning machines. Neural Netw 33:58–66

    MATH  Google Scholar 

  32. Lu C, Kao L (2016) A clustering-based sales forecasting scheme by using extreme learning machine and ensembling linkage methods with applications to computer server. Eng Appl Artif Intel 55:231–238

    Google Scholar 

  33. Lu S, Zhang G, Wang X (2012) A rank reduced matrix method in extreme learning machine. In: J. Wang, G. Yen, M. Polycarpou (eds.) Advances in Neural Networks-ISNN 2012, pp. 72–79

  34. López-Guede J, Izquierdo A, Estévez J, Graña M (2021) Active learning for road lane landmark inventory with V-ELM in highly uncontrolled image capture conditions. Neurocomputing 438:259–269

    Google Scholar 

  35. Mai Z, Chen Y, Du L (2021) A novel blind mmwave channel estimation algorithm based on ML-ELM. IEEE Commun Lett 25(5):1549–1553

    Google Scholar 

  36. Mao W, Zheng Y, Mu X, Zhao J (2014) Uncertainty evaluation and model selection of extreme learning machine based on Riemannian metric. Neur Comput Appl 24:1613–1625

    Google Scholar 

  37. Merkel C, Kudithipudi D (2014) Neuromemristive extreme learning machines for pattern classification. In: 2014 IEEE Comput Soc Ann Sympos on VLSI, pp. 77–82

  38. Ouyang T (2021) Feature learning for stacked ELM via low-rank matrix factorization. Neurocomputing 448:82–93

    Google Scholar 

  39. Perales-González C, Carbonero-Ruz M, Pérez-Rodríguez J, Becerra-Alonso D, Fernández-Navarro F (2020) Negative correlation learning in the extreme learning machine framework. Neural Comput Appl 32(17):13805–13823

    Google Scholar 

  40. Strassen V (1969) Gaussian elimination is not optimal. Numer Math 13:354–356

    MathSciNet  MATH  Google Scholar 

  41. Surantha N, Lesmana T, Isa S (2021) Sleep stage classification using extreme learning machine and particle swarm optimization for healthcare big data. J Big Data 8(1):1–17

    Google Scholar 

  42. Wan Y, Song S, Huang G, Li S (2017) Twin extreme learning machines for pattern classification. Neurocomputing 260:235–244

    Google Scholar 

  43. Wang H, He Q, Shang T, Zhuang F, Shi Z (2015) Extreme learning machine ensemble classifier for large-scale data. In: Proc ELM-2014, vol. 1, pp. 151–161. Springer

  44. Wang Y, Cao F, Yuan Y (2011) A study on effectiveness of extreme learning machine. Neurocomputing 74:2483–2490

    Google Scholar 

  45. Xin J, Wang Z, Qu L, Wang G (2019) Elastic extreme learning machine for big data classification. Neurocomputing 149:464–471

    Google Scholar 

  46. Xing Y, Ban X, Liu X, Shen Q (2019) Large-scale traffic congestion prediction based on the symmetric extreme learning machine cluster fast learning method. Symmetry 11(6(730)):1–19

    Google Scholar 

  47. Zhai J, Shao Q, Wang X (2016) Architecture selection of ELM networks based on sensitivity of hidden nodes. Neural Process Lett 44(2):471–489

    Google Scholar 

  48. Zhai J, Wang J, Wang X (2014) Ensemble online sequential extreme learning machine for large data set classification. In: Proc IEEE Intl Conf Syst Man Cyb, pp. 2250–2255

  49. Zhai J, Zhang S, Wang C (2017) The classification of imbalanced large data sets based on MapReduce and ensemble of ELM classifiers. Intl J Mach Learn Cyber 8:1009–1017

    Google Scholar 

  50. Zhu S, Wang H, Lv H, Zhang H (2021) Augmented online sequential quaternion extreme learning machine. Neural Process Lett 53:1161–1186

    Google Scholar 

  51. Zong W, Huang G, Chen Y (2013) Weighted extreme learning machine for imbalance learning. Neurocomputing 101:229–242

    Google Scholar 

Download references

Acknowledgements

This work has received financial support from the Consellería de Educación, Universidade e Formación Profesional (accreditation 2019–2022 ED431G-2019/04) and the European Regional Development Fund (ERDF), which acknowledges the CiTIUS—Centro Singular de Investigación en Tecnoloxías Intelixentes da Universidade de Santiago de Compostela as a Research Center of the Galician University System.

Author information

Authors and Affiliations

  1. Centro Singular de Investigación en Tecnoloxías Intelixentes da USC (CiTIUS), University of Santiago de Compostela, 15782, Santiago de Compostela, Spain

    Audi Albtoush, Manuel Fernández-Delgado, Eva Cernadas & Senén Barro

Authors
  1. Audi Albtoush
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Manuel Fernández-Delgado
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Eva Cernadas
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Senén Barro
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Manuel Fernández-Delgado.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Albtoush, A., Fernández-Delgado, M., Cernadas, E. et al. Quick extreme learning machine for large-scale classification. Neural Comput & Applic 34, 5923–5938 (2022). https://doi.org/10.1007/s00521-021-06727-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-021-06727-8

Keywords

Search

Navigation