Skip to main content
SpringerLink
Log in

Deep Weighted Extreme Learning Machine

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

The imbalanced data classification attracts increasing attention in the past years due to the continuous expansion of data available in many areas, such as biomedical engineering, surveillance, and computer vision. Learning from imbalanced data is challenging as most standard algorithms fail to properly represent the inherent complex characteristics of the data distribution. As an emerging technology, the extreme learning machine (ELM) and its variants, including the weighted ELM (WELM) and the boosting weighted ELM (BWELM), have been recently developed for the classification of imbalanced data. However, the WELM suffers the following deficiencies: (i) the sample weight matrix is manually chosen and fixed during the learning phase; (ii) the representation capability, namely the capability to extract features or useful information from the original data, is insufficiently explored due to its shallow structure. The BWELM employs the AdaBoost algorithm to optimize the sample weights. But the representation capability is still restricted by the shallow structure. To alleviate these deficiencies, we propose a novel deep weighted ELM (DWELM) algorithm for imbalanced data classification in this paper. An enhanced stacked multilayer deep representation network trained with the ELM (EH-DrELM) is first proposed to improve the representation capability, and a fast AdaBoost algorithm for imbalanced multiclass data (AdaBoost-ID) is developed to optimize the sample weights. Then, the novel DWELM for the imbalance learning is obtained by combining the above two algorithms. Experimental results on nine imbalanced binary-class datasets, nine imbalanced multiclass datasets, and five large benchmark datasets (three for multiclass and two for binary-class) show that the proposed DWELM achieves a better performance than the WELM and BWELM, as well as several state-of-the-art multilayer network-based learning algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. The datasets can be downloaded at http://sci2s.ugr.es/keel/imbalanced.php

  2. The datasets can be downloaded at http://www.ics.uci.edu/mlearn/MLRepository.html

  3. http://featureselection.asu.edu/datasets.php

  4. http://www.ics.uci.edu/mlearn/MLRepository.html

References

  1. Zong W, Huang G-B, Chen Y. Weighted extreme learning machine for imbalance learning. Neurocomputing 2013;101:229–242.

    Article  Google Scholar 

  2. Yu W, Zhuang F, He Q, Shi Z. Learning deep representations via extreme learning machines. Neurocomputing 2015;149:308–315.

    Article  Google Scholar 

  3. Li K, Kong X, Lu Z, Liu W, Yin J. Boosting weighted ELM for imbalanced learning. Neurocomputing 2014;128(5):15–21.

    Article  Google Scholar 

  4. Vinyals O, Jia Y, Deng L, Darrell T. Learning with recursive perceptual representations. Neural Information Processing Systems 2012;15:2834–2842.

    Google Scholar 

  5. Friedman J, Hastie T, Ti bshirani R. Additive logistic regression: a statistical view of boosting. Ann Stat 2000;28(2):337–407.

    Article  Google Scholar 

  6. Jiang Y, Shen Y, Liu Y, Liu W. Multiclass AdaBoost ELM and Its Application in LBP Based Face Recognition. Math Probl Eng 2015;2015:1–9.

    CAS  Google Scholar 

  7. Xia S, Meng F, Liu B, Zhou Y. A Kernel Clustering-Based Possibilistic Fuzzy Extreme Learning Machine for Class Imbalance Learning. Cogn Comput 2015;1(1):74–85.

    Article  Google Scholar 

  8. Savitha R, Suresh S, Kim HJ. A meta-cognitive learning algorithm for an extreme learning machine classifier. Cogn Comput 2014;6(2):253–263.

    Article  Google Scholar 

  9. Vong C, Ip W, Chiu C, Wong P. Imbalanced learning for air pollution by meta-cognitive online sequential extreme learning machine. Cogn Comput 2015;7(3):381–391.

    Article  Google Scholar 

  10. Chamara L, Zhou H, Huang G-B, Vong C-M. Representational learning with extreme learning machine for big data. IEEE Intell Syst 2013;28(6):31–34.

    Google Scholar 

  11. Tang J, Deng C, Huang G-B. Extreme learning machine for multilayer perceptron. IEEE Trans Neural Netw Learn Syst 2016;27(4):809–821.

    Article  PubMed  Google Scholar 

  12. Wong CM, Vong C, Wong P, Cao J. Kernel-based multilayer extreme learning machines for representation learning. IEEE Trans Neural Netw Learn Syst 2018;29(3):757–762.

    Article  PubMed  Google Scholar 

  13. Huang G-B, Zhu Q-Y, Siew C-K. Extreme learning machine: theory and applications. Neurocomputing 2006;70(1-3):489–501.

    Article  Google Scholar 

  14. Huang G-B, Zhou H, Ding X, Zhang R. Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern 2012;42(2):513–529.

    Article  Google Scholar 

  15. Deng W, Zheng Q, Chen L. 2009. Regularized extreme learning machine. In: IEEE Symposium on Computational Intelligence and Data Mining, pp 389–395,

  16. Huang G-B, Chen L, Siew C-K. Universal approximation using incremental constructive feedforward networks with random hidden nodes. Neural Netw Jul. 2006;17(4):879–892.

  17. Cao J, Zhang K, Luo M, Yin C, Lai X. Extreme learining machine and adaptive sparse representation for image classification. Neural Netw 2016;81:91–102.

    Article  PubMed  Google Scholar 

  18. Cao J, Hao J, Lai X, Vong C-M, Luo M. Ensemble extreme learning machine and sparse representation classification algorithm. J Frankl Inst 2016;353(17):4526–4541.

    Article  Google Scholar 

  19. Zhang R, Lan Y, Huang G-B, Xu Z, Soh Y. Dynamic extreme learning machine and its approximation capability. IEEE Trans Cybern 2013;43(6):2054–2065.

    Article  PubMed  Google Scholar 

  20. Li S, You ZH, Guo H, Luo X, Zhao ZQ. Inverse-free extreme learning machine with optimal information updating. IEEE Trans Cybern 2016;46(5):1229–1241.

    Article  PubMed  Google Scholar 

  21. Chen B, Xing L, Wang X, Qin J, Zheng N. Robust learning with kernel mean p-power error loss. IEEE Transactions on Cybernetics 2017;PP(99):1–13.

    Google Scholar 

  22. Chen B, Zhao S, Zhu P, Principe J. Quantized kernel recursive least squares algorithm. IEEE Trans Neural Netw Learn Syst 2013;24(9):1484–1491.

    Article  PubMed  Google Scholar 

  23. Nan S, Sun L, Chen B, Lin Z, Toh K. Density-dependent quantized least squares support vector machine for large data sets. IEEE Trans Neural Netw Learn Syst 2017;28(1):94–106.

    Article  PubMed  Google Scholar 

  24. Xu L, Ding S, Xu X, Zhang N. Self-adaptive extreme learning machine optimized by rough set theory and affinity propagation clustering. Cogn Comput 2016;8(4):720–728.

    Article  Google Scholar 

  25. Mohammed AA, Minhas R, Wu QMJ, Sid-Ahmed MA. Human face recognition based on multidimensional PCA and extreme learning machine. Pattern Recogn 2011;44(10-11):2588–2597.

    Article  Google Scholar 

  26. Chen T, Yap KH. Discriminative BoW framework for mobile landmark recognition. IEEE Trans Cybern 2014; 44(5):695–706.

    Article  PubMed  Google Scholar 

  27. Iosifidis A, Tefas A, Pitas I. Minimum class variance extreme learning machine for human action recognition. IEEE Trans Circ Syst Video Technol 2013;23(11):1968–1979.

    Article  Google Scholar 

  28. Cao J, Wang W, Wang J, Wang R. Excavation equipment recognition based on novel acoustic statistical features. IEEE Trans Cybern 2017;47(12):4392–4404.

    Article  PubMed  Google Scholar 

  29. Cao J, Shang L, Wang J, Vong C, Yin C, Cheng Y, Huang X. A novel distance estimation algorithm for periodic surface vibrations based on frequency band energy percentage feature. Mech Syst Signal Process 2018;113:222–236.

    Article  Google Scholar 

  30. Wang S, Deng C, Lin W, Huang GB, Zhao B. NMF-Based image quality assessment using extreme learning machine. IEEE Trans Cybern 2017;47(1):232–243.

    Article  PubMed  Google Scholar 

  31. Deng C, Wang S, Li Z, Huang G-B, Lin W. 2017. Content-insensitive blind image blurriness assessment using Weibull statistics and sparse extreme learning machine, IEEE Transactions on Systems, Man, and Cybernetics: Systems. https://doi.org/10.1109/TSMC.2017.2718180.

  32. Liu Y, Vong C, Wong P. Extreme learning machine for huge hypotheses re-ranking in statistical machine translation. Cogn Comput 2017;9(2):285–294.

    Article  Google Scholar 

  33. Duan L, Bao M, Cui S, Qiao Y, Miao J. Motor imagery EEG classification based on kernel hierarchical extreme learning machine. Cogn Comput 2017;9(6):758–765.

    Article  Google Scholar 

  34. Wang J, Ye K, Cao J, Wang T, Xue A, Cheng Y, Yin C. DOA estimation of excavation devices with ELM and MUSIC-based Hybrid Algorithm. Cogn Comput 2017;9(4):564–580.

    Article  Google Scholar 

  35. Cao J, Cao M, Wang J, Yin C, Wang D, Vidal P. 2018. Urban noise recognition with convolutional neural network. Multimedia Tools and Applications. https://doi.org/10.1007/s11042-018-6295-8.

  36. Chou C, Kuo B, Chang F. 2006. The generalized condensed nearest neighbor rule as a data reduction method. In: 18th International Conference on Pattern Recognition, pp 556–559.

  37. Kubat M, Matwin S. 1997. Addressing the curse of imbalanced training sets: one-sided selection. International Conference on Machine Learning.

  38. Laurikkala J. Improving identification of difficult small classes by balancing CLASS distribution. Artificial Intelligence in Medicine 2001;2101:63–66.

    Article  Google Scholar 

  39. Tang Y, Zhang YQ, Chawla NV, Krasser S. SVMS modeling for highly imbalanced classification. IEEE Trans Syst Man Cybern Part B (Cybern) 2009;39(1):281–288.

    Article  Google Scholar 

  40. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: synthetic minority over-sampling technique. J Artif Intell Res 2002;16(1):321–357.

    Article  Google Scholar 

  41. Han H, Wang W, Mao B. Borderline-SMOTE: a new over-sampling method in imbalanced data sets learning. Int Conf Intell Comput 2005;3644:878–887.

    Google Scholar 

  42. Cervantes J, Li X, Yu W. Using genetic algorithm to improve classification accuracy on imbalanced data. 2013 IEEE International Conference on Systems, Man, and Cybernetics, Manchester; 2013. p. 2659–2664.

  43. Sun Y, Kamel MS, Wong AK, Wang Y. Cost-sensitive boosting for classification of imbalanced data. Pattern Recogn 2007;40(12):3358–3378.

    Article  Google Scholar 

  44. Zhou Z, Liu X. Training cost-sensitive neural networks with methods addressing the class imbalance problem. IEEE Trans Knowl Data Eng 2006;18(1):63–77.

    Article  Google Scholar 

  45. Gao M, Hong X, Chen S, Harris CJ. A combined SMOTE and PSO based RBF classifier for two-class imbalanced problems. Neurocomputing 2011;74(17):3456–3466.

    Article  Google Scholar 

  46. Hong X, Chen S, Harris CJ. A kernel-based two-class classifier for imbalanced data sets. IEEE Trans Neural Netw 2007;18(1):28–41.

    Article  PubMed  Google Scholar 

  47. Mathew J, Luo M, Pang CK, Chan HL. Kernel-based SMOTE for SVM classification of imbalanced datasets. In: IECON 2015 - 41st Annual Conference of the IEEE Industrial Electronics Society; 2015. p. 1127–1132.

  48. He H, Garcia EA. Learning from imbalanced data. IEEE Trans Knowl Data Eng 2009;21(9):1263–1284.

    Article  Google Scholar 

  49. Cohen G, Hilario M, Sax H, Hugonnet S, Geissbuhler A. Learning from imbalanced data in surveillance of nosocomial infection. Artif Intell Med 2006;37(1):7–18.

    Article  PubMed  Google Scholar 

  50. Rao RB, Krishnan S, Niculescu RS. Data mining for improved cardiac care. ACM SIGKDD Explor Newsl 2006;8(1):3–10.

    Article  Google Scholar 

  51. Yu C, Chou L, Chang D. Predicting protein-protein interactions in unbalanced data using the primary structure of proteins. BMC Bioinforma 2010;11(1):167–177.

    Article  CAS  Google Scholar 

  52. Provost F, Fawcett T, Kohavi R. The case against accuracy estimation for comparing induction algorithms. In: Proceedings of the 15th International Conference on Machine Learning; 1998. p. 445–453.

  53. Fawcett T, Provost F. Adaptive fraud detection. Data Min Knowl Disc 1997;1:291–316.

    Article  Google Scholar 

  54. Cieslak DA, Chawla NV, Striegel A. Combating imbalance in network intrusion datasets. In: Proceedings of the 2006 IEEE International Conference on Granular Computing; 2006. p. 732–737.

  55. Weiss GM, Hirsh H. Learning to predict rare events in event sequences. In: Proceedings of the Fourth IEEE International Conference on Knowledge Discovery and Data Mining; 1998. p. 359–363.

  56. Mao W, Wang J, Wang L. Online sequential classification of imbalanced data by combining extreme learning machine and improved SMOTE algorithm. In: 2015 International Joint Conference on Neural Networks; 2015. p. 1–6.

  57. Shukla S, Yadav RN. Regularized weighted circular complex-valued extreme learning machine for imbalanced learning. IEEE Access 2015;3:3048–3057.

    Article  Google Scholar 

  58. Sharma R, Bist AS. Genetic algorithm based weighted extreme learning machine for binary imbalance learning. In: 2015 International Conference on Cognitive Computing and Information Processing; 2015. p. 1-6.

  59. Ren F, Cao P, Li W, Zhao D, Zaiane O. Ensemble based adaptive over-sampling method for imbalanced data learning in computer aided detection of microaneurysm. Computerized Medical Imaging and Graphics. 2017.

  60. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); 2016. p. 770-778.

  61. Achlioptas D. Database-friendly random projections. In: 20th ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems; 2001. p. 274–281.

  62. Hinton G, Salakhutdinov R. Reducing the dimensionality of data with neural networks. Science 2006;313 (5786):504–507.

    Article  CAS  PubMed  Google Scholar 

  63. Hinton G, Salakhutdinov R. Reducing the dimensionality of data with neural networks. Science 2006;313 (5786):504–507.

    Article  CAS  PubMed  Google Scholar 

  64. Krizhevsky A, Sutskever I, Hinton G. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems. 2012.

Download references

Funding

This work was supported by the NNSF of China (61503104, 61573123, 91648208), the 973 Program under Grant 2015CB351703, and the support of K. C. Wong Education Foundation and DAAD.

Author information

Authors and Affiliations

  1. Key Lab for IOT and Information Fusion Technology of Zhejiang, Hangzhou Dianzi University, Zhejiang, 310018, China

    Tianlei Wang, Jiuwen Cao & Xiaoping Lai

  2. Institute of Information and Control, Hangzhou Dianzi University, Zhejiang, 310018, China

    Jiuwen Cao & Xiaoping Lai

  3. School of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an, 710049, China

    Badong Chen

Authors
  1. Tianlei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiuwen Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoping Lai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Badong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jiuwen Cao.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Informed Consent

Informed consent was not required as no human or animals were involved.

Human and Animal Rights

This article does not contain any studies with human or animal subjects performed by any of the authors.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, T., Cao, J., Lai, X. et al. Deep Weighted Extreme Learning Machine. Cogn Comput 10, 890–907 (2018). https://doi.org/10.1007/s12559-018-9602-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-018-9602-9

Keywords

Search

Navigation