Skip to main content
Springer Nature Link
Log in

Neural Networks Saturation Reduction

  • Conference paper
  • First Online:
Artificial Intelligence and Soft Computing (ICAISC 2018)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 10841))

Included in the following conference series:

  • 2411 Accesses

Abstract

The saturation of particular neuron and a whole neural network is one of the reasons for problems with training effectiveness. The paper shows neural network saturation analysis, proposes a method for detection of saturated neurons and its reduction to achieve better training performance. The proposed approach has been confirmed by several experiments.

This work was supported by the National Science Centre, Krakow, Poland, undergrant No.2015/17/B/ST6/01880.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Rozycki, P., Kolbusz, J., Wilamowski, B.M.: Dedicated deep neural network architectures and methods for their training. In: IEEE 19th International Conference on Intelligent Engineering Systems (INES 2015) Bratislava, 3–5 September 2015, pp. 73–78 (2015)

    Google Scholar 

  2. Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. 18, 1527–1554 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  3. Mohamed, A., Dahl, G.E., Hinton, G.E.: Acoustic modeling using deep belief networks. IEEE Trans. Audio Speech Lang. Process. 20, 14–22 (2012)

    Article  Google Scholar 

  4. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

    Google Scholar 

  5. Simonyan K., Zisserman A.: Very deep convolutional networks for largescale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  6. Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)

    Article  Google Scholar 

  7. Silver, D., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)

    Article  Google Scholar 

  8. Wilamowski, B.M., Yu, H.: Neural network learning without backpropagation. IEEE Trans. Neural Netw. 21(11), 1793–1803 (2010)

    Article  Google Scholar 

  9. Hunter, D., Hao, Y., Pukish, M.S., Kolbusz, J., Wilamowski, B.M.: Selection of proper neural network sizes and architectures—A comparative study. IEEE Trans. Industr. Inf. 8, 228–240 (2012)

    Article  Google Scholar 

  10. Hochreiter, S.: The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertain. Fuzz. Knowl. Based Syst. 06, 107 (1998)

    Article  MATH  Google Scholar 

  11. Larochelle, H., et al.: Exploring strategies for training deep neural networks. J. Mach. Learn. Res. 10(Jan), 1–40 (2009)

    MATH  Google Scholar 

  12. He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: ICCV (2015)

    Google Scholar 

  13. Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is dificult. IEEE Trans. Neural Netw. 5(2), 157–166 (1994)

    Article  Google Scholar 

  14. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: International Conference on Articial Intelligence and Statistics, pp. 249–256 (2010)

    Google Scholar 

  15. Lee, C.Y., Xie, S., Gallagher, P., Zhang, Z., Tu, Z.: Deeply-supervised nets. arXiv preprint arXiv:1409.5185 (2014)

  16. Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML (2015)

    Google Scholar 

  17. Srivastava, R.K., Greff, K., Schmidhuber, J.: Highway networks. arXiv preprint arXiv:1505.00387 (2015)

  18. Kolbusz J., Różycki P., Wilamowski B.M.: The study of architecture MLP with linear neurons in order to eliminate the “vanishing gradient” problem. In: Artificial Intelligence and Soft Computing, ICAISC 2017, pp. 97–106 (2017)

    Google Scholar 

  19. Rakitianskaia, A., Engelbrecht, A.: Measuring saturation in neural networks. In: 2015 IEEE Symposium Series on Computational Intelligence, Cape Town, pp. 1423–1430 (2015)

    Google Scholar 

  20. LeCun, Y.A., Bottou, L., Orr, G.B., Müller, K.-R.: Efficient backprop. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. LNCS, vol. 7700, pp. 9–48. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35289-8_3

    Chapter  Google Scholar 

  21. Rakitianskaia, A., Engelbrecht, A.: Training high-dimensional neural networks with cooperative particle swarm optimiser. In: 2014 International Joint Conference on Neural Networks (IJCNN), Beijing, pp. 4011–4018 (2014)

    Google Scholar 

  22. Wilamowski, B.M., Yu, H.: Improved computation for levenberg Marquardt training. IEEE Trans. Neural Netw. 21(6), 930–937 (2010)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Information Technology and Management in Rzeszow, Rzeszów, Poland

    Janusz Kolbusz & Pawel Rozycki

  2. National Technical University of Ukraine “Igor Sikorsky Kyiv Polytechnic Institute”, Kiev, Ukraine

    Oleksandr Lysenko

  3. Auburn University, Auburn, AL, 36849-5201, USA

    Bogdan M. Wilamowski

Authors
  1. Janusz Kolbusz
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pawel Rozycki
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Oleksandr Lysenko
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bogdan M. Wilamowski
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pawel Rozycki .

Editor information

Editors and Affiliations

  1. Częstochowa University of Technology, Częstochowa, Poland

    Leszek Rutkowski

  2. Częstochowa University of Technology, Częstochowa, Poland

    Rafał Scherer

  3. Częstochowa University of Technology, Częstochowa, Poland

    Marcin Korytkowski

  4. University of Alberta, Edmonton, AB, Canada

    Witold Pedrycz

  5. AGH University of Science and Technology, Kraków, Poland

    Ryszard Tadeusiewicz

  6. University of Louisville, Louisville, KY, USA

    Jacek M. Zurada

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kolbusz, J., Rozycki, P., Lysenko, O., Wilamowski, B.M. (2018). Neural Networks Saturation Reduction. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J. (eds) Artificial Intelligence and Soft Computing. ICAISC 2018. Lecture Notes in Computer Science(), vol 10841. Springer, Cham. https://doi.org/10.1007/978-3-319-91253-0_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-91253-0_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-91252-3

  • Online ISBN: 978-3-319-91253-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics