Abstract
The performance of artificial neural networks significantly depends on the choice of the nonlinear activation function of the neuron. Usually this choice comes down to an empirical one from a list of universal functions that have shown satisfactory results on most tasks. However this approach does not lead to optimal training in terms of model convergence over a certain number of epochs. We proposed tunable polynomial activation function for artificial neuron. Parameters of this function can be adjusted during learning procedure along with synaptic weights. The proposed function can take the form of universal ones due to its polynomial properties. Adjustable form tunable polynomial function leads to the fastest convergence of the model and more accurate training due to the possibility of using a smaller training step that has been shown experimentally. Improved convergence allows to apply tunable activation function to various deep learning problems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Agostinelli, F., Hoffman, M., Sadowski, P., Baldi, P.: Learning activation functions to improve deep neural networks. arXiv preprint arXiv:1412.6830 (2014)
Bodyanskiy, Y., Kulishova, N., Rudenko, O.: One model of formal neuron. Rep. Nat. Acad. Sci. Ukraine 4, 69–73 (2001)
Bottou, L., Bousquet, O.: The tradeoffs of large scale learning. In: Advances in Neural Information Processing Systems, pp. 161–168 (2008)
Cichocki, A., Unbehauen, R., Swiniarski, R.W.: Neural Networks for Optimization and Signal Processing, vol. 253. Wiley, New York (1993). https://doi.org/10.1016/0925-2312(94)90041-8
Clevert, D.A., Unterthiner, T., Hochreiter, S.: Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 (2015)
Cybenko, G.: Approximation by superpositions of a sigmoidal function. Math. Control Signals Systems 2(4), 303–314 (1989). https://doi.org/10.1007/BF02134016
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, pp. 249–256 (2010)
Glorot, X., Bordes, A., Bengio, Y.: Deep sparse rectifier neural networks. In: In: Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, pp. 315–323 (2011)
Heaton, Y., Jeff, I.G., Bengio, Y., Courville, A.: Deep learning. Genetic programming and evolvable machines. Nature 19(1–2), 305–307 (2017). https://doi.org/10.1007/s10710-017-9314-z
Goyal, M., Goyal, R., Lall, B.: Learning activation functions: a new paradigm of understanding neural networks. arXiv preprint arXiv:1906.09529 (2019)
Graupe, D.: Deep Learning Neural Networks: Design and Case Studies. World Scientific Publishing Company (2016)
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015). https://doi.org/10.1109/iccv.2015.123
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016). https://doi.org/10.1109/cvpr.2016.90
Hendrycks, D., Gimpel, K.: Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 (2016)
Hornik, K.: Approximation capabilities of multilayer feedforward networks. Neural Netw. 4(2), 251–257 (1991). https://doi.org/10.1016/0893-6080(91)90009-t
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images. Technical report, University of Toronto (2009)
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015). https://doi.org/10.1038/nature14539
LeCun, Y., et al.: Backpropagation applied to handwritten zip code recognition. Neural Comput. 1(4), 541–551 (1989). https://doi.org/10.1162/neco.1989.1.4.541
LeCun, Y., Cortes, C., Burges, C.: MNIST handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist (2010)
Lu, L., Shin, Y., Su, Y., Karniadakis, G.E.: Dying relu and initialization: theory and numerical examples. arXiv preprint arXiv:1903.06733 (2019). https://doi.org/10.4208/cicp.OA-2020-0165
Lu, Z., Pu, H., Wang, F., Hu, Z., Wang, L.: The expressive power of neural networks: a view from the width. In: Advances in Neural Information Processing Systems, pp. 6231–6239 (2017)
Molina, A., Schramowski, P., Kersting, K.: Pade activation units: end-to-end learning of flexible activation functions in deep networks. arXiv preprint arXiv:1907.06732 (2019)
Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: ICML (2010)
Schmidhuber, J.: Deep learning in neural networks: an overview. Neural Netw. 61, 85–117 (2015). https://doi.org/10.1016/j.neunet.2014.09.003
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Sonoda, S., Murata, N.: Neural network with unbounded activation functions is universal approximator. Appl. Comput. Harmonic Anal. 43(2), 233–268 (2017). https://doi.org/10.1016/j.acha.2015.12.005
Arai, K., Kapoor, S., Bhatia, R. (eds.): SAI 2020. AISC, vol. 1230. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-52243-8
Xu, B., Wang, N., Chen, T., Li, M.: Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853 (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Bilonoh, B., Bodyanskiy, Y., Kolchygin, B., Mashtalir, S. (2022). Tunable Activation Functions for Deep Neural Networks. In: Babichev, S., Lytvynenko, V. (eds) Lecture Notes in Computational Intelligence and Decision Making. ISDMCI 2021. Lecture Notes on Data Engineering and Communications Technologies, vol 77. Springer, Cham. https://doi.org/10.1007/978-3-030-82014-5_43
Download citation
DOI: https://doi.org/10.1007/978-3-030-82014-5_43
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-82013-8
Online ISBN: 978-3-030-82014-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)