Skip to main content
Springer Nature Link
Log in

Estimation of Deep Neural Networks Capabilities Using Polynomial Approach

  • Conference paper
  • First Online:
Artificial Intelligence and Soft Computing (ICAISC 2016)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 9692))

Included in the following conference series:

Abstract

Currently very popular trend in artificial intelligence is the use of deep neural networks. The power of such networks are very large, but the main difficulty is learning these networks. The article presents a analysis of deep neural network nonlinearity with polynomial approximation of neuron activation functions. It is shown that nonlinearity grows exponentially with the depth of the neural network. The effectiveness of the approach is demonstrated by several experiments.

This work was supported by the National Science Centre, Cracow, Poland under Grant No. 2013/11/B/ST6/01337.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Deng, L., Li, X.: Machine learning paradigms in speech recognition: An overview. IEEE Trans. Audio Speech Lang. Process. 21(5), 1060–1089 (2013)

    Article  Google Scholar 

  2. Lee, H., Grosse, R., Ranganath, R., Ng, A.: Unsupervised learning of hierarchical representations with convolutional deep belief networks. Commun. Assoc. Comput. Mach. (ACM) 54(10), 95–103 (2011)

    Google Scholar 

  3. Wilamowski, B.M.: Neural network architectures and learning algorithms - how not to be frustrated with neural networks. IEEE Ind. Electron. Mag. 3(4), 56–63 (2009)

    Article  Google Scholar 

  4. Hunter, D., Yu, H., Pukish, M.S., Kolbusz, J., Wilamowski, B.M.: Selection of proper neural network sizes and architectures–A comparative study. IEEE Trans. Industr. Inf. 8, 228–240 (2012)

    Article  Google Scholar 

  5. Bengio, Y.: Learning deep architectures for AI. Found. Trends Mach. Learn. 2(1), 1–127 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  6. Hochreiter, S.: The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Unc. Fuzz. Knowl. Based Syst. 06, 107 (1998)

    Article  MATH  Google Scholar 

  7. Erhan, D., Bengio, Y., Courvelle, A., Manzagol, P., Vencent, P., Bengio, S.: Why does unsupervised pre-training help deep learning? J. Mach. Learn. Res. 9, 201–208 (2010)

    MathSciNet  MATH  Google Scholar 

  8. Schmidhuber, J.: Deep learning in neural networks: An overview. Neural Netw. 61, 85–117 (2015)

    Article  Google Scholar 

  9. LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)

    Article  Google Scholar 

  10. Olas, T., Mleczko, W.K., Nowicki, R.K., Wyrzykowski, R., Krzyzak, A.: Adaptation of RBM learning for intel MIC architecture. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) Artificial Intelligence and Soft Computing. LNCS, vol. 9119, pp. 90–101. Springer, Heidelberg (2015)

    Chapter  Google Scholar 

  11. Wilamowski, B.M., Hunter, D., Malinowski, A.: Solving parity-N problems with feedforward neural networks. In: Proceedings of the 2003 IEEE IJCNN, pp. 2546–2551. IEEE Press (2003)

    Google Scholar 

  12. Xu, Y.: An experimental study on speech enhancement based on deep neural networks. IEEE Signal Process. Lett. 21(1), 65–68 (2011)

    Article  Google Scholar 

  13. Gao, J., et al.: Modeling interestingness with deep neural networks. U.S. Patent No. 20,150,363,688, 17 December 2015

    Google Scholar 

  14. Erhan, D., et al.: Scalable object detection using deep neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014)

    Google Scholar 

  15. Zhang, X., et al.: Improving deep neural network acoustic models using generalized maxout networks. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE (2014)

    Google Scholar 

  16. Montufar, G.F., et al.: On the number of linear regions of deep neural networks. In: Advances in Neural Information Processing Systems (2014)

    Google Scholar 

  17. Oquab, M., et al.: Learning and transferring mid-level image representations using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2014)

    Google Scholar 

  18. Szegedy, C., Toshev, A., Erhan, D.: Deep neural networks for object detection. In: Advances in Neural Information Processing Systems (2013)

    Google Scholar 

  19. Wilamowski, B.M., Yu, H.: Neural network learning without backpropagation. IEEE Trans. Neural Netw. 21(11), 1793–1803 (2010)

    Article  Google Scholar 

  20. Wilamowski, B.M., Yu, H.: NNT - Neural Networks Trainer. http://www.eng.auburn.edu/wilambm/nnt/. Accessed 15 January 2016

  21. Alvarez-Sanchez, J.R.: Injecting knowledge into the solution of the two-spiral problem. Neural Comput. Appl. 8, 265–272 (1999)

    Article  Google Scholar 

  22. Różycki, P., Kolbusz, J., Bartczak, T., Wilamowski, B.M.: Using parity-N problems as a way to compare abilities of shallow, very shallow and very deep architectures. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) Artificial Intelligence and Soft Computing. LNCS, vol. 9119, pp. 112–122. Springer, Heidelberg (2015)

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Information Technology and Management in Rzeszow, Sucharskiego 2, 35-225, Rzeszow, Poland

    Pawel Rozycki, Janusz Kolbusz & Roman Korostenskyi

  2. Auburn University, Auburn, AL, 36849-5201, USA

    Bogdan M. Wilamowski

Authors
  1. Pawel Rozycki
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Janusz Kolbusz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Roman Korostenskyi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bogdan M. Wilamowski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pawel Rozycki .

Editor information

Editors and Affiliations

  1. Częstochowa University of Technology, Czestochowa, Poland

    Leszek Rutkowski

  2. Częstochowa University of Technology, Czestochowa, Poland

    Marcin Korytkowski

  3. Częstochowa University of Technology, Czestochowa, Poland

    Rafał Scherer

  4. AGH University of Science and Technology, Krakow, Poland

    Ryszard Tadeusiewicz

  5. University of California, Berkeley, California, USA

    Lotfi A. Zadeh

  6. University of Louisville, Louisville, Kentucky, USA

    Jacek M. Zurada

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this paper

Cite this paper

Rozycki, P., Kolbusz, J., Korostenskyi, R., Wilamowski, B.M. (2016). Estimation of Deep Neural Networks Capabilities Using Polynomial Approach. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L., Zurada, J. (eds) Artificial Intelligence and Soft Computing. ICAISC 2016. Lecture Notes in Computer Science(), vol 9692. Springer, Cham. https://doi.org/10.1007/978-3-319-39378-0_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-39378-0_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-39377-3

  • Online ISBN: 978-3-319-39378-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics