Abstract
In the last decades, deep learning has led to spectacular successes. One of the reasons for these successes was the fact that deep neural networks use a special Rectified Linear Unit (ReLU) activation function \(s(x)=\max (0,x)\). Why this activation function is so successful is largely a mystery. In this paper, we show that common sense ideas—as formalized by fuzzy logic—can explain this mysterious effectiveness.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Belohlavek, R., Dauben, J.W., Klir, G.J.: Fuzzy Logic and Mathematics: a Historical Perspective. Oxford University Press, New York (2017)
Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, New York (2006)
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge, Massachusetts (2016)
Klir, G., Yuan, B.: Fuzzy Sets and Fuzzy Logic. Prentice Hall, Upper Saddle River, New Jersey (1995)
Mendel, J.M.: Uncertain Rule-Based Fuzzy Systems: introduction and New Directions. Springer, Cham, Switzerland (2017)
Nguyen, H.T., Walker, C.L., Walker, E.A.: A First Course in Fuzzy Logic. Chapman and Hall/CRC, Boca Raton, Florida (2019)
Novák, V., Perfilieva, I., Močkoř, J.: Mathematical Principles of Fuzzy Logic. Kluwer, Boston, Dordrecht (1999)
Zadeh, L.A.: Fuzzy sets. Inf. Control 8, 338–353 (1965)
Acknowledgements
This work was supported in part by the National Science Foundation grants 1623190 (A Model of Change for Preparing a New Generation for Professional Practice in Computer Science), HRD-1834620 and HRD-2034030 (CAHSI Includes), EAR-2225395, and by the AT &T Fellowship in Information Technology. It was also supported by the program of the development of the Scientific-Educational Mathematical Center of Volga Federal District No. 075-02-2020-1478, and by a grant from the Hungarian National Research, Development and Innovation Office (NRDI).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Urenda, J., Kosheleva, O., Kreinovich, V. (2024). Fuzzy Techniques Explain the Effectiveness of ReLU Activation Function in Deep Learning. In: Castillo, O., Melin, P. (eds) New Horizons for Fuzzy Logic, Neural Networks and Metaheuristics. Studies in Computational Intelligence, vol 1149. Springer, Cham. https://doi.org/10.1007/978-3-031-55684-5_28
Download citation
DOI: https://doi.org/10.1007/978-3-031-55684-5_28
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-55683-8
Online ISBN: 978-3-031-55684-5
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)