Skip to main content
SpringerLink
Log in

Multi-criteria decision making based architecture selection for single-hidden layer feedforward neural networks

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

Architecture selection is a fundamental problem in artificial neural networks, which could be treated as a decision making process that evaluates, ranks, and makes choices from a set of network structures. Traditional methods evaluate a network structure by designing a criterion based on a validation model or an error bound model. On one hand, the time complexity of a validation model is usually high; on the other hand, different validation models or error bound models may lead to different (even conflicting) results, which post challenges to the traditional single criterion-based architecture selection methods. In the area of decision making, many problems employed multiple criteria since the performance is better than using a single criterion. In this paper, we propose a multi-criteria decision making based architecture selection algorithm for single-hidden layer feedforward neural networks trained by extreme learning machine. Two criteria are incorporated into the selection process, i.e., training accuracy and the Q-value estimated by the localized generalization error model. The training accuracy reflects the capability of the model on correctly categorizing the known samples, and the Q-value estimated by localized generalization error model reflects the size of the neighbourhood of training samples in which the model can predict unseen samples with confidence. By achieving a trade-off between these two criteria, a new architecture selection algorithm is proposed. Experimental comparisons demonstrate the feasibility and effectiveness of the proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Haykin S (1998) Neural networks: a comprehensive foundation. Prentice Hall, Upper Saddle River

    MATH  Google Scholar 

  2. Ham FM, Kostanic I (2000) Principles of neurocomputing for science and engineering. McGraw-Hill Higher Education, New York

    Google Scholar 

  3. Huang G-B, Zhu Q-Y, Siew C-K (2004) Extreme learning machine: a new learning scheme of feedforward neural networks, vol 2. In: Proceedings. 2004 IEEE international joint conference on neural networks, IEEE, Budapest, pp 985–990

  4. Huang G-B, Zhou H, Ding X, Zhang R (2012) Extreme learning machine for regression and multiclass classification. IEEE Trans Syst Man Cybern Part B Cybern 42(2):513–529

    Article  Google Scholar 

  5. Wang R, Kwong S, Wang X (2012) A study on random weights between input and hidden layers in extreme learning machine. Soft Comput 16(9):1465–1475

    Article  Google Scholar 

  6. Wang R, Kwong S, Wang DD (2013) An analysis of ELM approximate error based on random weight matrix. Int J Uncertain Fuzziness Knowl Based Syst 21(supp02):1–12

    Article  MathSciNet  MATH  Google Scholar 

  7. Huang G, Huang G-B, Song S, You K (2015) Trends in extreme learning machines: a review. Neural Netw 61(1):32–48

    Article  MATH  Google Scholar 

  8. Liu Y (1995) Unbiased estimate of generalization error and model selection in neural network. Neural Netw 8(2):215–219

    Article  Google Scholar 

  9. Zhang S, McCullagh P, Nugent C, Zheng H, Baumgarten M (2011) Optimal model selection for posture recognition in home-based healthcare. Int J Mach Learn Cybern 2(1):1–14

    Article  Google Scholar 

  10. Kapanova KG, Dimov I, Sellier JM (2016) A genetic approach to automatic neural network architecture optimization. Neural Comput Appl. https://doi.org/10.1007/s00521-016-2510-6 (in press)

  11. Chhachhiya D, Sharma A, Gupta M (2017) Designing optimal architecture of neural network with particle swarm optimization techniques specifically for educational dataset. In: Proceedings. 7th international conference on cloud computing, data science and engineering—confluence, IEEE, Noida, pp 52–57

  12. Aras S, Kocakoc ID (2016) A new model selection strategy in time series forecasting with artificial neural networks: IHTS. Neurocomputing 174:974–987

    Article  Google Scholar 

  13. Ciancio C, Ambrogio G, Gagliardi F, Musmanno R (2016) Heuristic techniques to optimize neural network architecture in manufacturing applications. Neural Comput Appl 27:2001–2015

    Article  Google Scholar 

  14. Silva AJ, Ludermir TB, Oliveira WR (2016) Quantum perceptron over a field and neural network architecture selection in a quantum computer. Neural Netw 76:55–64

    Article  Google Scholar 

  15. Silva AJ, Oliveira WR, Ludermir TB (2016) Weightless neural network parameters and architecture selection in a quantum computer. Neurocomputing 183:13–22

    Article  Google Scholar 

  16. Rosli N, Ibrahim R, Ismail I, Hassan SM, Chung TD (2016) Neural network architecture selection for efficient prediction model of gas metering system. In: Proceedings. 2nd IEEE international symposium on robotics and manufacturing automation, IEEE, Ipoh, pp 1–5

  17. Curtis P, Harb M, Abielmona R, Petriu E (2016) Feature selection and neural network architecture evaluation for real-time video object classification. In: Proceedings. 2016 IEEE congress on evolutionary computation, IEEE, Vancouver, BC, pp 1038–1045

  18. Yeung DS, Ng WWY, Wang D, Tsang ECC, Wang X-Z (2007) Localized generalization error model and its application to architecture selection for radial basis function neural network. IEEE Trans Neural Netw 18(5):1294–1305

    Article  Google Scholar 

  19. Yeung DS, Patrick PKC, Ng WWY (2009) Radial basis function network learning using localized generalization error bound. Inf Sci 179(19):3199–3217

    Article  MATH  Google Scholar 

  20. Polhill GJ, Weir MK (2001) An approach to guaranteeing generalisation in neural networks. Neural Netw 14(8):1035–1048

    Article  Google Scholar 

  21. Wang X-Z, Shao Q-Y, Miao Q, Zhai J-H (2013) Architecture selection for networks trained with extreme learning machine using localized generalization error model. Neurocomputing 102:3–9

    Article  Google Scholar 

  22. Shen D, Zhang J, Su J, Zhou G, Tan CL (2004) Multi-criteria-based active learning for named entity recognition. In: Proceedings. 42nd annual meeting on association for computational linguistics, Association for Computational Linguistics, p 589

  23. Chang TH, Wang TC (2009) Using the fuzzy multi-criteria decision making approach for measuring the possibility of successful knowledge management. Inf Sci 179(4):355–370

    Article  Google Scholar 

  24. Figueira J, Greco S, Ehrgott M (2005) Multiple criteria decision analysis: state of the art surveys. Springer, Berlin

    Book  MATH  Google Scholar 

  25. Fonseca CM, Fleming PJ (1993) Genetic algorithms for multiobjective optimization: formulation, discussion and generalization, vol 1. In: Proceedings. 5th international conference on genetic algorithms, San Mateo, pp 416

  26. Branke J, Kaußler T, Schmeck H (2001) Guidance in evolutionary multi-objective optimization. Adv Eng Softw 32(6):499–507

    Article  MATH  Google Scholar 

  27. Srinivas N, Deb K (1994) Muiltiobjective optimization using nondominated sorting in genetic algorithms. Evol Comput 2(3):221–248

    Article  Google Scholar 

  28. Corne D, Knowles J, Oates M (2000) The pareto envelope-based selection algorithm for multiobjective optimization. In: Parallel problem solving from nature. Springer, Berlin, pp 839–848

  29. Roy B, Slowinski R (1993) Criterion of distance between technical programming and socio-economic priority. RAIRO Oper Res 27(1):45–60

    Article  MathSciNet  MATH  Google Scholar 

  30. Xu X, Martel JM, Lamond BF (2001) A multiple criteria ranking procedure based on distance between partial preorders. Eur J Oper Res 133(1):69–80

    Article  MathSciNet  MATH  Google Scholar 

  31. Jabeur K, Martel JM, Khélifa SB (2004) A distance-based collective preorder integrating the relative importance of the group’s members. Group Decis Negot 13(4):327–349

    Article  Google Scholar 

  32. Wang R, Kwong S (2014) Active learning with multi-criteria decision making systems. Pattern Recognit 47(9):3106–3119

    Article  MATH  Google Scholar 

  33. Hoeffding W (1963) Probability inequalities for sums of bounded random variables. J Am Stat Assoc 58(301):13–30

    Article  MathSciNet  MATH  Google Scholar 

  34. Ng WWY, Yeung DS, Wang X-Z, Cloete I (2004) A study of the difference between partial derivative and stochastic neural network sensitivity analysis for applications in supervised pattern classification problems. In: Proceedings. 2004 international conference on machine learning and cybernetics, vol 7. IEEE, Shanghai, pp 4283–4288

  35. Wang X-Z, Li C-G, Yeung DS, Song S, Feng H (2008) A definition of partial derivative of random functions and its application to RBFNN sensitivity analysis. Neurocomputing 71(7):1515–1526

    Article  Google Scholar 

  36. Campanella G, Ribeiro RA (2011) A framework for dynamic multiple-criteria decision making. Decis Support Syst 52(1):52–60

    Article  Google Scholar 

  37. Hipel KW, Radford KJ, Fang L (1993) Multiple participant-multiple criteria decision making. IEEE Trans Syst Man Cybern 23(4):1184–1189

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 61772344, Grant 61402460, Grant 61732011, and Grant 61472257, in part by the Natural Science Foundation of SZU under Grant 2017060, in part by the Guangdong Provincial Science and Technology Plan Project under Grant 2013B040403005, in part by the HD Video R&D Platform for Intelligent Analysis and Processing in Guangdong Engineering Technology Research Centre of Colleges and Universities under Grant GCZX-A1409, in part by the Internal Research Grant (RG 66/2016-2017) of The Education University of Hong Kong, and in part by a grant from Research Grants Council of Hong Kong Special Administrative Region, China (UGC/FDS11/E04/16).

Author information

Authors and Affiliations

  1. College of Mathematics and Statistics, Shenzhen University, Shenzhen, 518060, China

    Ran Wang, Jiqiang Feng & Chen Xu

  2. Department of Mathematics and Information Technology, The Education University of Hong Kong, Hong Kong, Hong Kong

    Haoran Xie

  3. Caritas Institute of Higher Education, Hong Kong, Hong Kong

    Fu Lee Wang

Authors
  1. Ran Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haoran Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiqiang Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Fu Lee Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chen Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ran Wang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, R., Xie, H., Feng, J. et al. Multi-criteria decision making based architecture selection for single-hidden layer feedforward neural networks. Int. J. Mach. Learn. & Cyber. 10, 655–666 (2019). https://doi.org/10.1007/s13042-017-0746-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-017-0746-9

Keywords

Search

Navigation