Skip to main content
SpringerLink
Log in

Extreme learning machine for structured output spaces

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Recently, extreme learning machine (ELM) has attracted increasing attention due to its successful applications in classification, regression, and ranking. Normally, the desired output of the learning system using these machine learning techniques is a simple scalar output. However, there are many applications in machine learning which require more complex output rather than a simple scalar one. Therefore, structured output is used for such applications where the system is trained to predict structured output instead of simple one. Previously, support vector machine (SVM) has been introduced for structured output learning in various applications. However, from machine learning point of view, ELM is known to offer better generalization performance compared to other learning techniques. In this study, we extend ELM to more generalized framework to handle complex outputs where simple outputs are considered as special cases of it. Besides the good generalization property of ELM, the resulting model will possesses rich internal structure that reflects task-specific relations and constraints. The experimental results show that structured ELM achieves similar (for binary problems) or better (for multi-class problems) generalization performance when compared to ELM. Moreover, as verified by the simulation results, structured ELM has comparable or better precision performance with structured SVM when tested for more complex output such as object localization problem on PASCAL VOC2006. Also, the investigation on parameter selections is presented and discussed for all problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Barron AR (1993) Universal approximation bounds for superpositions of a sigmoidal function. IEEE Trans Inf Theory 39:930–945

    Article  MathSciNet  MATH  Google Scholar 

  2. Leshno M, Lin VY, Pinkus A, Schocken S (1993) Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Netw 6:861–867

    Article  Google Scholar 

  3. Huang G-B, Zhu Q-Y, Siew C-K (2006) Extreme learning machine: theory and applications. Neurocomputing 70:489–501

    Article  Google Scholar 

  4. Huang G-B, Zhu QY, Siew C-K (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: Proceedings of the 2004 IEEE international joint conference on neural networks, pp 985–990

  5. Huang G-B, Chen L, Siew C-K (2006) Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans Neural Netw 17:879–892

    Article  Google Scholar 

  6. Huang G-B, Chen L (2007) Convex incremental extreme learning machine. Neurocomputing 70:3056–3062

    Article  Google Scholar 

  7. Huang G-B, Chen L (2008) Enhanced random search based incremental extreme learning machine. Neurocomputing 71:3460–3468

    Article  Google Scholar 

  8. Huang G-B, Ding X, Zhou H (2010) Optimization method based extreme learning machine for classification. Neurocomputing 74:155–163

    Article  Google Scholar 

  9. Huang G-B, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern B Cybern 42:513–529

    Article  Google Scholar 

  10. Liang N-Y, Huang G-B, Saratchandran P, Sundararajan N (2006) A fast and accurate online sequential learning algorithm for feedforward networks. IEEE Trans Neural Netw 17:1411–1423

    Article  Google Scholar 

  11. Huang G-B, Siew C-K (2004) Extreme learning machine: RBF network case. In: 8th control, automation, robotics and vision conference (ICARCV 2004), pp 1029–1036

  12. Huang G-B, Zhu Q-Y, Mao K, Siew C-K, Saratchandran P, Sundararajan N (2006) Can threshold networks be trained directly? IEEE Trans Circuits Syst II Express Briefs 53:187–191

    Article  Google Scholar 

  13. Huang G-B, Siew C-K (2006) Real-time learning capability of neural networks. IEEE Trans Neural Netw 17:863–878

    Article  Google Scholar 

  14. Ding S, Xu X, Nie R (2014) Extreme learning machine and its applications. Neural Comput Appl 25:549–556

    Article  Google Scholar 

  15. Frénay B, Verleysen M (2010) Using SVMs with randomised feature spaces: an extreme learning approach. In: ESANN

  16. Frénay B, Verleysen M (2011) Parameter-insensitive kernel in extreme learning for non-linear support vector regression. Neurocomputing 74:2526–2531

    Article  Google Scholar 

  17. Fu H, Vong C-M, Wong P-K, Yang Z (2016) Fast detection of impact location using kernel extreme learning machine. Neural Comput Appl 27:121–130

    Article  Google Scholar 

  18. Rong H-J, Ong Y-S, Tan A-H, Zhu Z (2008) A fast pruned-extreme learning machine for classification problem. Neurocomputing 72:359–366

    Article  Google Scholar 

  19. Sun Z-L, Choi T-M, Au K-F, Yu Y (2008) Sales forecasting using extreme learning machine with applications in fashion retailing. Decis Support Syst 46:411–419

    Article  Google Scholar 

  20. Zong W, Huang G-B (2014) Learning to rank with extreme learning machine. Neural Process Lett 39:155–166

    Article  Google Scholar 

  21. Schapire WWCRE, Singer Y (1998) Learning to order things. Adv Neural Inf Process Syst 10:451

    Google Scholar 

  22. Ailon N, Mohri M (2008) An efficient reduction of ranking to classification. In: Proceedings of the 21st Conference on Computational Learning Theory, COLT, pp 87–98

  23. Joachims T, Finley T, Yu C-NJ (2009) Cutting-plane training of structural SVMs. Mach Learn 77:27–59

    Article  MATH  Google Scholar 

  24. BakIr G (2007) Predicting structured data. MIT Press, Cambridge

    Google Scholar 

  25. Blaschko MB, Lampert CH (2008) Learning to localize objects with structured output regression. In: Computer vision—ECCV 2008. Springer, Berlin, pp 2–15

  26. Hare S, Saffari A, Torr PH (2011) Struck: structured output tracking with kernels. In: IEEE international conference on computer vision (ICCV), pp 263–270

  27. Tsochantaridis I, Hofmann T, Joachims T, Altun Y (2004) Support vector machine learning for interdependent and structured output spaces. In: Proceedings of the 21st international conference on machine learning, p 104

  28. Schulz H, Behnke S (2014) Structured prediction for object detection in deep neural networks. In: Artificial neural networks and machine learning—ICANN 2014. Springer, Berlin, pp 395–402

  29. Annema AJ, Hoen K, Wallinga H (1994) Precision requirements for single-layer feedforward neural networks. In: Proceedings of the Fourth International Conference on Microelectronics for Neural Networks and Fuzzy Systems, Turin, pp 145–151

  30. Huang G-B, Babri H (1998) Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions. IEEE Trans Neural Netw 9:224–229

    Article  Google Scholar 

  31. Huang G-B (2003) Learning capability and storage capacity of two-hidden-layer feedforward networks. IEEE Trans Neural Netw 14:274–281

    Article  Google Scholar 

  32. Rao CR, Mitra SK (1971) Generalized inverse of matrices and its applications, vol 7. Wiley, New York

    MATH  Google Scholar 

  33. Roller BTCGD (2004) Max-margin Markov networks. Adv Neural Inf Process Syst 16:25

    Google Scholar 

  34. Taskar B, Lacoste-Julien S, Jordan M (2005) Structured prediction via the extragradient method. In: NIPS

  35. Tsochantaridis I, Joachims T, Hofmann T, Altun Y (2005) Large margin methods for structured and interdependent output variables. J Mach Learn Res 6:1453–1484

    MathSciNet  MATH  Google Scholar 

  36. Everingham M, Zisserman A., Williams CKI, Van Gool L (2006) The pascal visual object classes (voc) challenge . Int J Comput Vis 88:303. doi: 10.1007/s11263-009-0275-4

    Article  Google Scholar 

  37. Lampert CH, Blaschko MB, Hofmann T (2008) Beyond sliding windows: object localization by efficient subwindow search. In: IEEE conference on computer vision and pattern recognition (CVPR 2008), pp 1–8

  38. Bay H, Tuytelaars T, Van Gool L (2006) Surf: speeded up robust features. In: Computer vision—ECCV 2006. Springer, Berlin, pp 404–417

Download references

Acknowledgements

This work is financially supported by Fundamental Research Grant Scheme (FRGS), VOTE 4F331 from Ministry of Higher Education, Malaysia.

Author information

Authors and Affiliations

  1. Centre for Artificial Intelligence and Robotics, Malaysia-Japan International Institute of Technology, Universiti Teknologi Malaysia, Kuala Lumpur, Malaysia

    Ayman Maliha, Rubiyah Yusof & Mohd Ibrahim Shapiai

Authors
  1. Ayman Maliha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rubiyah Yusof
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohd Ibrahim Shapiai
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ayman Maliha.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Maliha, A., Yusof, R. & Shapiai, M.I. Extreme learning machine for structured output spaces. Neural Comput & Applic 30, 1251–1264 (2018). https://doi.org/10.1007/s00521-016-2754-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-016-2754-1

Keywords

Search

Navigation