Skip to main content
Springer Nature Link
Log in

Two Hidden Layers are Usually Better than One

  • Conference paper
  • First Online:
Engineering Applications of Neural Networks (EANN 2017)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 744))

  • 3677 Accesses

Abstract

This study investigates whether feedforward neural networks with two hidden layers generalise better than those with one. In contrast to the existing literature, a method is proposed which allows these networks to be compared empirically on a hidden-node-by-hidden-node basis. This is applied to ten public domain function approximation datasets. Networks with two hidden layers were found to be better generalisers in nine of the ten cases, although the actual degree of improvement is case dependent. The proposed method can be used to rapidly determine whether it is worth considering two hidden layers for a given problem.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw. 2, 359–366 (1989)

    Article  Google Scholar 

  2. Hornik, K., Stinchcombe, M., White, H.: Some new results on neural network approximation. Neural Netw. 6, 1069–1072 (1993)

    Article  Google Scholar 

  3. Huang, G.-B., Babri, H.A.: Upper bounds on the number of hidden neurons in feedforward networks with arbitrary bounded nonlinear activation functions. IEEE Trans. Neural Netw. 9, 224–229 (1998)

    Article  Google Scholar 

  4. Zhang, G.P.: Avoiding pitfalls in neural network research. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 37, 3–16 (2007)

    Article  Google Scholar 

  5. Chester, D.L.: Why two hidden layers are better than one. In: Caudhill, M. (ed.) International Joint Conference on Neural Networks, vol. 1, pp. 265–268. Laurence Erlbaum, New Jersey (1990)

    Google Scholar 

  6. Brightwell, G., Kenyon, C., Paugam-Moisy, H.: Multilayer neural networks: one or two hidden layers? In: Mozer, M.C., Jordan, M.I., Petsche, T. (eds.) Advances in Neural Information Processing Systems, vol. 9, pp. 148–154. MIT Press, Cambridge (1997)

    Google Scholar 

  7. Sontag, E.D.: Feedback stabilization using two-hidden-layer nets. IEEE Trans. Neural Netw. 3, 981–990 (1992)

    Article  Google Scholar 

  8. Thomas, A.J., Walters, S.D., Petridis, M., Malekshahi Gheytassi, S., Morgan, R.E.: Accelerated optimal topology search for two-hidden-layer feedforward neural networks. In: Jayne, C., Iliadis, L. (eds.) EANN 2016. CCIS, vol. 629, pp. 253–266. Springer, Cham (2016). doi:10.1007/978-3-319-44188-7_19

    Chapter  Google Scholar 

  9. Thomas, A.J., Walters, S.D., Malekshahi Gheytassi, S., Morgan, R.E., Petridis, M.: On the optimal node ratio between hidden layers: a probabilistic study. Int. J. Mach. Learn. Comput. 6, 241–247 (2016). doi:10.18178/ijmlc.2016.6.5.605

    Article  Google Scholar 

  10. Nakama, T.: Comparisons of single- and multiple-hidden-layer neural networks. In: Liu, D., Zhang, H., Polycarpou, M., Alippi, C., He, H. (eds.) Advances in Neural Networks – ISNN 2011 Part 1. LNCS, vol. 6675, pp. 270–279. Springer, Heidelberg (2011)

    Chapter  Google Scholar 

  11. Funahashi, K.-I.: On the approximate realization of continuous mappings by neural networks. Neural Netw. 2, 183–192 (1989)

    Article  Google Scholar 

  12. Idler, C.: Pattern recognition and machine learning techniques for algorithmic trading. MA thesis, FernUniversität, Hagen, Germany (2014)

    Google Scholar 

  13. Moré, J.J.: The Levenberg-Marquardt algorithm: implementation and theory. In: Watson, G.A. (ed.) Numerical Analysis. LNM, vol. 630, pp. 105–116. Springer, Heidelberg (1978). doi:10.1007/BFb0067700

    Chapter  Google Scholar 

  14. Beale, M.H., Hagan, M.T., Demuth, H.B.: Neural Network Toolbox User’s guide. https://www.mathworks.com/help/pdf_doc/nnet/nnet_ug.pdf

  15. UCI Machine Learning Repository. https://archive.ics.uci.edu/ml/

  16. Bilkent University Function Approximation Repository. http://funapp.cs.bilkent.edu.tr/DataSets/

  17. Regression Datasets. http://www.dcc.fc.up.pt/~ltorgo/Regression/DataSets.html

  18. Yeh, I.-C.: Modeling of strength of high performance concrete using artificial neural networks. Cem. Concr. Res. 28, 1797–1808 (1998)

    Article  Google Scholar 

Download references

Acknowledgements

We thank Prof. Martin T. Hagan of Oklahoma State University for kindly donating the Engine dataset used in this paper to Matlab. Thanks also to Prof. I-Cheng Yeh for permission to use his Concrete Compressive Strength dataset [18], as well as the other donors of the various datasets used in this study.

Author information

Authors and Affiliations

  1. School of Computing Engineering and Mathematics, University of Brighton, Brighton, UK

    Alan J. Thomas, Simon D. Walters, Saeed Malekshahi Gheytassi & Robert E. Morgan

  2. Faculty of Science and Technology, Middlesex University, London, UK

    Miltos Petridis

Authors
  1. Alan J. Thomas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Miltos Petridis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Simon D. Walters
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Saeed Malekshahi Gheytassi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Robert E. Morgan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alan J. Thomas .

Editor information

Editors and Affiliations

  1. Politecnico di Milano, Milan, Italy

    Giacomo Boracchi

  2. Democritus University of Thrace, University Campus, Xanthi, Greece

    Lazaros Iliadis

  3. School of Computing Science and Digital Media, Robert Gordon University, Aberdeen, United Kingdom

    Chrisina Jayne

  4. Univesity of Ioannina, Ioannina, Greece

    Aristidis Likas

Appendix A – Full Results: Average Node for Node Comparisons

Appendix A – Full Results: Average Node for Node Comparisons

See Figs. 4, 5 and 6.

Fig. 4.
figure 4

Abalone (top), Airfoil, Chemical and Concrete (bottom)

Full size image
Fig. 5.
figure 5

Delta Elevators (top), Engine, Kinematics, and Mortgage (bottom)

Full size image
Fig. 6.
figure 6

Simplefit (top) and White Wine (bottom)

Full size image

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Thomas, A.J., Petridis, M., Walters, S.D., Gheytassi, S.M., Morgan, R.E. (2017). Two Hidden Layers are Usually Better than One. In: Boracchi, G., Iliadis, L., Jayne, C., Likas, A. (eds) Engineering Applications of Neural Networks. EANN 2017. Communications in Computer and Information Science, vol 744. Springer, Cham. https://doi.org/10.1007/978-3-319-65172-9_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-65172-9_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-65171-2

  • Online ISBN: 978-3-319-65172-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics