Skip to main content
SpringerLink
Log in

Fisher-regularized supervised and semi-supervised extreme learning machine

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

The structural information of data contains useful prior knowledge and thus is important for designing classifiers. Extreme learning machine (ELM) has been a potential technique in handling classification problems. However, it only simply considers the prior class-based structural information and ignores the prior knowledge from statistics and geometry of data. In this paper, to capture more structural information of the data, we first propose a Fisher-regularized extreme learning machine (called Fisher-ELM) by applying Fisher regularization into the ELM learning framework, the main goals of which is to build an optimal hyperplane such that the output weight and within-class scatter are minimized simultaneously. The proposed Fisher-ELM reflects both the global characteristics and local properties of samples. Intuitively, the Fisher-ELM can approximatively fulfill the Fisher criterion and can obtain good statistical separability. Then, we exploit graph structural formulation to obtain semi-supervised Fisher-ELM version (called Lap-FisherELM) by introducing manifold regularization that characterizes the geometric information of the marginal distribution embedded in unlabeled samples. An efficient successive overrelaxation algorithm is used to solve the proposed Fisher-ELM and Lap-FisherELM, which converges linearly to a solution, and can process very large datasets that need not reside in memory. The proposed Fisher-ELM and Lap-FisherELM do not need to deal with the extra matrix and burden the computations related to the variable switching, which makes them more suitable for relatively large-scale problems. Experiments on several datasets verify the effectiveness of the proposed methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Notes

  1. http://archive.ics.uci.edu/ml/datasets.html.

  2. http://www.cad.zju.edu.cn/home/dengcai/Data/MLData.html.

  3. https://www.mathworks.com/.

  4. http://www.cad.zju.edu.cn/home/dengcai/Data/MLData.html.

  5. http://www.cs.columbia.edu/CAVE/software/softlib/coil-20.php.

  6. https://cs.nyu.edu/~roweis/data.html.

References

  1. Huang GB, Chen YQ, Babri HA (2002) Classification ability of single hidden layer feedforward neural networks. IEEE Trans Neural Netw 11(3):799–801

    Article  Google Scholar 

  2. Rumelhart DE (1986) Learning representations by back-propagating errors. Nature 323:533–536

    Article  MATH  Google Scholar 

  3. Hagan MT, Menhaj MB (1994) Training feedforward networks with the marquardt algorithm. IEEE Trans Neural Netw 5(6):989–993

    Article  Google Scholar 

  4. Chng ES, Yang HH, Bos S (1996) Orthogonal least-squares learning algorithm with local adaptation process for the radial basis function networks. IEEE Signal Process Lett 3(8):253–255

    Article  Google Scholar 

  5. Huang G, Song S, Wu C (2012) Orthogonal least squares algorithm for training cascade neural networks. IEEE Trans Circuits Syst I Regul Pap 59(11):2629–2637

    Article  MathSciNet  Google Scholar 

  6. Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297

    MATH  Google Scholar 

  7. Vapnik Vladimir N (2002) The nature of statistical learning theory. IEEE Trans Neural Netw 8(6:1564–1564

    Google Scholar 

  8. Huang GB, Zhu QY, Siew CK (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: Proceedings 2004 IEEE International Joint Conference on Neural Networks. IEEE

  9. Huang GB, Zhu QY, Siew CK (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1):489–501

    Article  Google Scholar 

  10. Huang GB, Chen L, Siew CK (2006) Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans Neural Netw 17(4):879–892

    Article  Google Scholar 

  11. Wang Y, Cao F, Yuan Y (2011) A study on effectiveness of extreme learning machine. Neurocomputing 74(16):2483–2490

    Article  Google Scholar 

  12. Huang GB, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern Part B Cybern A Publ IEEE Syst Man Cybern Soc 42(2):513–529

    Article  Google Scholar 

  13. Wang Z, Zhao Y, Wang G (2017) Extreme learning machine for large-scale graph classification based on mapreduce. Neurocomputing 261:106–114

    Article  Google Scholar 

  14. Yang L, Zhang S (2016) A sparse extreme learning machine framework by continuous optimization algorithms and its application in pattern recognition. Eng Appl Artif Intell 53:176–189

    Article  Google Scholar 

  15. Wang GG, Lu M, Dong YQ, Zhao XJ (2016) Self-adaptive extreme learning machine. Neural Comput Appl 27(2):291–303

    Article  Google Scholar 

  16. Zhang Y, Wu J, Cai Z, Zhang P, Chen L (2016) Memetic extreme learning machine. Pattern Recogn 58(C):135–148

    Article  Google Scholar 

  17. Ding XJ, Chang BF (2014) Active set strategy of optimized extreme learning machine. Sci Bull 59(31):4152–4160

    Article  Google Scholar 

  18. Huang GB, Ding XJ, Zhou HM (2010) Optimization method based extreme learning machine for classification. Neurocomputing 74:155–163

    Article  Google Scholar 

  19. Davis JC (1996) Introduction to statistical pattern recognition. Comput Geosci 22(7):833–834

    Article  Google Scholar 

  20. Gupta H, Agrawal AK, Pruthi T, Shekhar C, Chellappa R (2002) An experimental evaluation of linear and kernel-based methods for face recognition. In: IEEE workshop on applications of computer vision. IEEE Computer Society, p 13

  21. Xiong T, Cherkassky V (2005) A combined SVM and LDA approach for classification. IEEE Int Joint Conf Neural Netw 3:1455–1459

    Google Scholar 

  22. Zafeiriou S, Tefas A, Pitas I (2007) Minimum class variance support vector machines. IEEE Trans Image Process A Publ IEEE Signal Process Soc 16(10):1651

    MathSciNet  Google Scholar 

  23. An W, Liang M (2013) Fuzzy support vector machine based on within-class scatter for classification problems with outliers or noises. Neurocomputing 110(6):101–110

    Article  Google Scholar 

  24. Zhang L, Zhou WD (2016) Fisher-regularized support vector machine. Inf Sci 343–344:79–93

    Article  MathSciNet  MATH  Google Scholar 

  25. Iosifidis A, Tefas A, Pitas I (2013) Minimum class variance extreme learning machine for human action recognition. IEEE Trans Circuits Syst Video Technol 23(11):1968–1979

    Article  Google Scholar 

  26. Iosifidis A, Tefas A, Pitas I (2015) Sparse extreme learning machine classifier exploiting intrinsic graphs. Elsevier, Amsterdam

    Book  Google Scholar 

  27. Iosifidis A, Tefas A, Pitas I (2016) Graph embedded extreme learning machine. IEEE Trans Cybern 46(1):311

    Article  Google Scholar 

  28. Belkin M, Niyogi P, Sindhwani V (2006) Manifold regularization: a geometric framework for learning from labeled and unlabeled examples. JMLR.org

  29. Chapelle O, Sindhwani V, Keerthi SS (2008) Optimization techniques for semi-supervised support vector machines. J Mach Learn Res 9(1):203–233

    MATH  Google Scholar 

  30. Melacci S, Belkin M (2009) Laplacian support vector machines trained in the primal. J Mach Learn Res 12(5):1149–1184

    MathSciNet  MATH  Google Scholar 

  31. Huang G, Song S, Gupta JND, Wu C (2014) Semi-supervised and unsupervised extreme learning machines. IEEE Trans Cybern 44(12):2405

    Article  Google Scholar 

  32. Iosifidis A, Tefas A, Pitas I (2014) Regularized extreme learning machine for multi-view semi-supervised action recognition. Neurocomputing 145(18):160–262

    Google Scholar 

  33. Zhou Y, Liu B, Xia S, Liu B (2015) Semi-supervised extreme learning machine with manifold and pairwise constraints regularization. Neurocomputing 149(PA):180–186

    Article  Google Scholar 

  34. Liu B, Xia SX, Meng FR, Zhou Y (2016) Manifold regularized extreme learning machine. Neural Comput Appl 27(2):255–269

    Article  Google Scholar 

  35. Pei H, Chen Y, Wu Y, Zhong P (2017) Laplacian total margin support vector machine based on within-class scatter. Knowl Based Syst 119:152–165

    Article  Google Scholar 

  36. Mangasarian OL, Musicant DR (1999) Successive overrelaxation for support vector machines. IEEE Trans Neural Netw 10(5):1032–7

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by National Natural Science Foundation of China (No11471010) and Chinese Universities Scientific Fund.

Author information

Authors and Affiliations

  1. College of Information and Electrical Engineering, China Agricultural University, Beijing, 100083, China

    Jun Ma & Yakun Wen

  2. College of Science, China Agricultural University, Beijing, 100083, China

    Liming Yang

Authors
  1. Jun Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yakun Wen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Liming Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Liming Yang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ma, J., Wen, Y. & Yang, L. Fisher-regularized supervised and semi-supervised extreme learning machine. Knowl Inf Syst 62, 3995–4027 (2020). https://doi.org/10.1007/s10115-020-01484-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-020-01484-x

Keywords

Search

Navigation